Change:
Still early in diagnosis, but soon after cutting 0.17.0, our early internal testing show this test as flaking. It's pretty bizarre, though, given the test isn't flaking on GCE, or even the continuous integration runs on GKE.
cc
So far, I've tried reproing this on a bare cluster 100 times, no joy. Now I'm trying up/test/down, no immediate joy (but it's a flake). If I get to the point of testing up/e2e/down, I might as well call it "Jenkins" and be done with it, but I guess it's time to start looking through logs instead. (As for the P0, trying to figure out if we need a cherry-pick, or if we let the flake proceed on in the entire 0.17.0 train.)
+1 to ironing out a flake; v0.16.0 had a flake (or maybe even a consistently-failing test) which made us turn off tests that would run against it for weeks. This is especially important now that we're doing soak tests.
(since it's related to events)
The test itself looks racy; there's nothing to guarantee that the events will have made it through the system by the time the pod is running. Generally one would expect that, but there's no guarantee.
Can you paste some sample failures here?
go/k8s-test/job/kubernetes-e2e-gke-test/4534/console
Actually this test is flaky across the board, and has been for some time. If you look at kubernetes-e2e-gce Test Results Analyser in Jenkins, you'll see about 8 failures in the past few hundred runs. I think lavalamp is correct - the test needs to retry/wait for the events to get through. Specifically, this code needs to be wrapped in a util.Poll():
Yeah if the kubelet event shows up less often than the scheduler event, that would make sense, since there'll have been less time.
I can whip up a quick fix now.
If it's a known flake, it's not going to block us. We should get a fix in soon for the next release, though.
Nearly done.

Discussion:
"github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega"

Failf("Failed to get pod: %v", err) } fmt.Printf("%+vn", podWithUid) <ins> var events *api.EventList </ins> // Check for scheduler event about the pod. By("checking for scheduler event about the pod") <del> events, err := c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUid.UID), "involvedObject.namespace": api.NamespaceDefault, "source":                   "scheduler", }.AsSelector(), ) if err != nil { Failf("Error while listing events: %v", err) } Expect(len(events.Items)).ToNot(BeZero(), "scheduler events from running pod") fmt.Println("Saw scheduler event for our pod.") </del> <ins> expectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { events, err := c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUid.UID), "involvedObject.namespace": api.NamespaceDefault, "source":                   "scheduler", }.AsSelector(), ) if err != nil { return false, err } if len(events.Items) > 0 { fmt.Println("Saw scheduler event for our pod.") return true, nil } return false, nil })) </ins> // Check for kubelet event about the pod. By("checking for kubelet event about the pod") <del> events, err = c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.uid":       string(podWithUid.UID), "involvedObject.kind":      "Pod", "involvedObject.namespace": api.NamespaceDefault, "source":                   "kubelet", }.AsSelector(), ) if err != nil { Failf("Error while listing events: %v", err) } Expect(len(events.Items)).ToNot(BeZero(), "kubelet events from running pod") fmt.Println("Saw kubelet event for our pod.") </del> <ins> expectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { events, err = c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.uid":       string(podWithUid.UID), "involvedObject.kind":      "Pod", "involvedObject.namespace": api.NamespaceDefault, "source":                   "kubelet", }.AsSelector(), ) if err != nil { return false, err } if len(events.Items) > 0 { fmt.Println("Saw kubelet event for our pod.") return true, nil } return false, nil })) </ins> }) })