Change:
log.Printf("Server version: %#vn", v) } <ins> for start := time.Now(); time.Since(start) < waitTimeout; time.Sleep(5 * time.Second) { eps := getWebserverEndpoints(client) if eps.Len() >= *peerCount { break } state.Logf("%v/%v has %v endpoints, which is less than %v as expected. Waiting for all endpoints to come up.", *namespace, *service, len(eps), *peerCount) } </ins> // Do this repeatedly, in case there's some propagation delay with getting // newly started pods into the endpoints list. for i := 0; i < 15; i++ { <del> endpoints, err := client.Endpoints(*namespace).Get(*service) if err != nil { state.Logf("Unable to read the endpoints for %v/%v: %v; will try again.", *namespace, *service, err) time.Sleep(time.Duration(1+rand.Intn(10)) * time.Second) } eps := sets.String{} for _, ss := range endpoints.Subsets { for _, a := range ss.Addresses { for _, p := range ss.Ports { eps.Insert(fmt.Sprintf("http://%s:%d", a.IP, p.Port)) } } } </del> <ins> eps := getWebserverEndpoints(client) </ins> for ep := range eps { state.Logf("Attempting to contact %s", ep) contactSingle(ep, state) } time.Sleep(5 * time.Second) } } <ins> //getWebserverEndpoints returns the webserver endpoints as a set of String, each in the format like "http://{ip}:{port}" func getWebserverEndpoints(client *client.Client) sets.String { endpoints, err := client.Endpoints(*namespace).Get(*service) eps := sets.String{} if err != nil { state.Logf("Unable to read the endpoints for %v/%v: %v.", *namespace, *service, err) return eps } for _, ss := range endpoints.Subsets { for _, a := range ss.Addresses { for _, p := range ss.Ports { eps.Insert(fmt.Sprintf("http://%s:%d", a.IP, p.Port)) } } } return eps } </ins> // contactSingle dials the address 'e' and tries to POST to its /write address. func contactSingle(e string, state *State) { body, err := json.Marshal(&WritePost{

Discussion:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The documentation suggests that "The default pull policy is IfNotPresent" This is not the case when imagePullPolicy is defined under the spec section in a Pod definition without a specific value. Example: This will result in a Pod having imagePullPolicy as Always. Why is this needed: This is needed for clarification of the default behaviour. It is necessary for the appropriate handling of helm charts, in cases that the imagePullPolicy is empty.
/sig docs
The imagePullPolicy default value is .
In this paragraph you can see that In your reference it points out the default action for the kubelet, not the behaviour when imagePullPolicy is declared. Please do have in mind that the imagePullPolicy option for Never is mentioned only in this paragraph in the document.
If an image tag of "latest" is used, the imagePullPolicy defaults to to "Always": If no tag and no digest are specified on the image, an image tag of "latest" is used: The behavior when no image digest or tag is specified should be documented in the API docs for the and fields:
I am interested on helping this issue, but I am new to kubernete repo, it could take me sometime
I found the following which is well documented in "Best Practices for Configuration" It would be expected to include more information on the options in the main link under
I'd like to work on this one.
Very welcome, but yesterday also mentioned that he is following up on this issue, you can communicate with him before starting

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...