Change:
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite} Previous issues for this test:
Most pods failed the predicate. do you mind taking a look since you modified the test to run in a large-cluster?
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
- I'm reassigning back to you since there are a bunch of other failures in small suites. I will debug the large cluster failure in the background.
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
Any updates? CC
The main problem is that the test gets a list of nodes at the beginning, and then creates a RC with N*10 pods, where N is the number of nodes. The test holds on to the initial node list and queries individual kubelet at the /runningPods endpoint to see all pods are running. Here comes the problem. For some reason, the initial list only includes 2 nodes, even though the cluster has 3 (non-master) nodes. The test did not check the pods (/runningPods) on the third node at all, and therefore can never find all pods created by the RC. Test creating 20 pods because it saw only two nodes, and it repeatedly checking those two nodes: The third node () actually existed and pods got scheduled on it: I am not sure yet why the test didn't get 3 nodes at the beginning, and whether the tests started before the cluster became ready.
The kubectl test "should remove all the taints with the same key off a node" adds taint to the node, causing the node to be temporarily ruled out in the cleanup test. I'll patch the cleanup test to avoid hitting this scenario, but I think the the kubectl test could interfere with other tests as well.
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}
https://k8s- Failed: [] kubelet [] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {Kubernetes e2e suite}

Discussion:
BeforeEach(func() { c = f.ClientSet <ins> // Use node labels to restrict the pods to be assigned only to the // nodes we observe initially. nodeLabels = make(map[string]string) nodeLabels["kubelet_cleanup"] = "true" </ins> nodes := framework.GetReadySchedulableNodesOrDie(c) numNodes = len(nodes.Items) nodeNames = sets.NewString() // If there are a lot of nodes, we don't want to use all of them // (if there are 1000 nodes in the cluster, starting 10 pods/node // will take ~10 minutes today). And there is also deletion phase. <del> // // Instead, we choose at most 10 nodes and will constraint pods // that we are creating to be scheduled only on that nodes. </del> <ins> // Instead, we choose at most 10 nodes. </ins> if numNodes > maxNodesToCheck { numNodes = maxNodesToCheck <del> nodeLabels = make(map[string]string) nodeLabels["kubelet_cleanup"] = "true" </del> } for i := 0; i < numNodes; i++ { nodeNames.Insert(nodes.Items[i].Name)