Change:
Probably not the same as ? To for triage
Shouldn't this fixed by ?
this has nothing to do with . kubelet is trying to create the mirror pod over and over again because it still hasn't observed the mirror pod from watch. It is probably the same as # .
I thought they might be different because this one has a bunch of "failed creating ... already exists" messages in it, but doesn't.
It is essentially the same problem -- the watch was delayed so kubelet kept seeing the old snapshot of the pods where the mirror didn't exist. That's why it's flaky and only happens on shippable (which I suspect has poorer performance) :( This becomes worse every time a new test is . I think we need to limit the number of concurrent tests in
limits the number of concurrent tests on travis/shippable to 4. Let's see if this helps reduce the timeout flakes.
Another example:
Here is yet another example: Retitled the issue since limiting the concurrent tests doesn't work. It could be that the server is heavily loaded and the test timed out, or there was a delay/drop in kubelet's pod watch stream.
Now example:
If, for whatever reason, a watch update for pod creation is dropped, kubelet would need to wait until re-listing to notice that. This would certainly cause the test to time out. would it be useful to shorten the re-listing interval for tests like this?
I am not quite sure how this works--let's chat about it tomorrow?
From , we can see that kubelet was lightly loaded as the update channel was mostly empty while waiting. This corroborates the theory that watch might be at fault.
cc: quinton-hoole
The WIP reflector benchmark () may shed some light on this issue once it's integrated with the continuous e2e performance suite. I will keep this issue open for now to gather more data (since this is not reproducible locally at all), and to see if there are other potential causes.
The PR above disables the static pod test temporarily so that we can track down the root cause without spamming people with the failures. Note that other tests in the same file () may still trigger the flaky failure, though less frequently.
Can this be closed yet?
No, integration test is still flaky. See for example. Many tests in rely on checking the pod status with a preset timeout. I disabled one test which failed most often, but other tests still fail from time to time.
The issue has been inactive for months, and there have not been any new reports. Closing.

Discussion:
} } <ins> // runStaticPodTest is disabled until #6651 is resolved. </ins> func runStaticPodTest(c *client.Client, configFilePath string) { var testCases = []struct { desc         string

firstManifestURL := ServeCachedManifestFile(testPodSpecFile) secondManifestURL := ServeCachedManifestFile(testManifestFile) <del> apiServerURL, configFilePath := startComponents(firstManifestURL, secondManifestURL, apiVersion) </del> <ins> apiServerURL, _ := startComponents(firstManifestURL, secondManifestURL, apiVersion) </ins> // Ok. we're good to go. glog.Infof("API Server started on %s", apiServerURL)

runSelfLinkTestOnNamespace(c, api.NamespaceDefault) runSelfLinkTestOnNamespace(c, "other") }, <del> func(c *client.Client) { runStaticPodTest(c, configFilePath) }, </del> } // Only run at most maxConcurrency tests in parallel.

//              1 pod infra container + 2 containers from the URL on first Kubelet + //              1 pod infra container + 2 containers from the URL on second Kubelet + //              1 pod infra container + 1 container from the service test. <del> // In addition, runStaticPodTest creates 2 pod infra containers + //                                       2 pod containers from the file (1 for manifest and 1 for spec) // The total number of container created is 13 </del> <ins> // The total number of container created is 9 </ins> <del> if len(createdConts) != 16 { glog.Fatalf("Expected 16 containers; got %vnnlist of created containers:nn%#vnnDocker 1 Created:nn%#vnnDocker 2 Created:nn%#vnn", len(createdConts), createdConts.List(), fakeDocker1.Created, fakeDocker2.Created) </del> <ins> if len(createdConts) != 12 { glog.Fatalf("Expected 12 containers; got %vnnlist of created containers:nn%#vnnDocker 1 Created:nn%#vnnDocker 2 Created:nn%#vnn", len(createdConts), createdConts.List(), fakeDocker1.Created, fakeDocker2.Created) </ins> } glog.Infof("OK - found created containers: %#v", createdConts.List())