Change:
err := waitForPodsCreatedRunningResponding(c, ns, name, replicas) Expect(err).NotTo(HaveOccurred(), "Each pod should start running and responding") <del> By("cause network partition on one node") nodelist, err := c.Nodes().List(labels.Everything(), fields.Everything()) </del> <ins> By("choose a node with at least one pod - we will block some network traffic on this node") label := labels.SelectorFromSet(labels.Set(map[string]string{"name": name})) pods, err := c.Pods(ns).List(label, fields.Everything()) // list pods after all have been scheduled </ins> Expect(err).NotTo(HaveOccurred()) <del> node := nodelist.Items[0] pod, err := waitForRCPodOnNode(c, ns, name, node.Name) Expect(err).NotTo(HaveOccurred()) Logf("Getting external IP address for %s", name) host := "" for _, a := range node.Status.Addresses { if a.Type == api.NodeExternalIP { host = a.Address + ":22" break } } Logf("Setting network partition on %s", node.Name) dropCmd := fmt.Sprintf("sudo iptables -I OUTPUT 1 -d %s -j DROP", testContext.CloudConfig.MasterName) if _, _, code, err := SSH(dropCmd, host, testContext.Provider); code != 0 || err != nil { Failf("Expected 0 exit code and nil error when running %s on %s, got %d and %v", dropCmd, node, code, err) } Logf("Waiting for node %s to be not ready", node.Name) waitForNodeToBe(c, node.Name, false, 2*time.Minute) </del> <ins> nodeName := pods.Items[0].Spec.NodeName </ins> <del> Logf("Waiting for pod %s to be removed", pod.Name) waitForRCPodToDisappear(c, ns, name, pod.Name) By("verifying whether the pod from the partitioned node is recreated") err = waitForPodsCreatedRunningResponding(c, ns, name, replicas) </del> <ins> node, err := c.Nodes().Get(nodeName) </ins> Expect(err).NotTo(HaveOccurred()) <del> // TODO: We should do this cleanup even if one of the above fails By("remove network partition") undropCmd := "sudo iptables --delete OUTPUT 1" if _, _, code, err := SSH(undropCmd, host, testContext.Provider); code != 0 || err != nil { Failf("Expected 0 exit code and nil error when running %s on %s, got %d and %v", undropCmd, node, code, err) } </del> <ins> By(fmt.Sprintf("block network traffic from node %s", node.Name)) performTemporaryNetworkFailure(c, ns, name, replicas, pods.Items[0].Name, node) </ins> Logf("Waiting for node %s to be ready", node.Name) waitForNodeToBe(c, node.Name, true, 2*time.Minute) By("verify wheter new pods can be created on the re-attached node") <del> err = resizeReplicationController(c, ns, name, replicas+1) </del> <ins> // increasing the RC size is not a valid way to test this // since we have no guarantees the pod will be scheduled on our node. additionalPod := "additionalpod" err = createServeHostnamePodOnNode(c, ns, additionalPod, node.Name) </ins> Expect(err).NotTo(HaveOccurred()) <del> err = waitForPodsCreatedRunningResponding(c, ns, name, replicas+1) Expect(err).NotTo(HaveOccurred()) _, err = waitForRCPodOnNode(c, ns, name, node.Name) </del> <ins> err = waitForPodsCreatedRunningResponding(c, ns, additionalPod, 1) </ins> Expect(err).NotTo(HaveOccurred()) <ins> // verify that it is really on the requested node { pod, err := c.Pods(ns).Get(additionalPod) Expect(err).NotTo(HaveOccurred()) if pod.Spec.NodeName != node.Name { Logf("Pod %s found on invalid node: %s instead of %s", pod.Spec.NodeName, node.Name) } } </ins> }) }) })

Discussion:
Master tracking bug:
The scenario of the test is the following: a service an RC with as many pods as the number of nodes for pods created, running and responding the first node and get the pod from it ... It is assumed that the first node has a pod - that's because kubernetes is supposed to distribute pods for which there is a service evenly among nodes. So we expect that each node has one pod. This turns out to be false. (in 80% cases the first node doesn't have a pod.
Thanks . cc: Is it a valid assumption that pods should be evenly spread across nodes? I assume not? If even spreading is a valid assumption, then there's a bug in the system. If not, then we need to fix this test.
To clarify the above, it's perfectly sensible to me that k8s should in an ideal world spread pods that constitute a service across as many nodes as possible. But I doubt whether our scheduler has such logic yet?
Our spreading is only a probabilistic guarantee. So this is incorrect test behavior.
We do actually have spreading logic. To determine if this is a scheduler bug, you'd have to know the state of the system more exactly. Spreading is balanced with resource usage, for example-- if something big was a little slow leaving the node, it could cause it to not get a pod. Or if it was briefly unavailable.
OK, I'd suggest that we change the test then, to not assume that pods are perfectly evenly spread across nodes.
, is there a way to run the scheduler with debug logs? I'm interested about the scores for nodes. (even if it won't help me in this issue, it's a nice exercise) I suppose the test fails because CPU utilization has a significant impact on scheduling pods - more that having a service. That's why all pods are scheduled on the same node. I'm going to remove the requirement that there is one pod per node (I'll still create a service though - just to make pods spread more evenly). I'll simply remove from the network one node that has a pod (without assuming that the first one has). I believe the scenario will still test everything we need ( please correct me if I'm wrong). The PR should be ready today.

By the way, a side effect of this test is that in a 2-node cluster, all add-ons will end up on one node, occupying 0.9 cores - this setup will stay for further tests, so it's good to be aware of it. I'll write this comment in the test code as well.
is ready for review. Possible second round will be needed. No changes in production code have been made (tests only).
I think if you set v=10 you'll see the log messages with that info in them.
This scenario is OK.
, sorry for such simple questions, but how do I set v=10?
you need to edit and change to in the command line args. The kubelet should then restart the container (validate with ) if it doesn't, should fix it.
Status: waiting for review of