Change:
cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.
i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely. I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...

Discussion:
"Rerun it with at least two nodes to get complete coverage.") } <del> podNames := LaunchNetTestPodPerNode(f, nodes, svcname, "1.7") </del> <ins> podNames := LaunchNetTestPodPerNode(f, nodes, svcname, "1.8") </ins> // Clean up the pods defer func() {

DoRaw() } <del> timeout := time.Now().Add(2 * time.Minute) </del> <ins> // nettest containers will wait for all service endpoints to come up for 2 minutes // apply a 3 minutes observation period here to avoid this test to time out before the nettest starts to contact peers timeout := time.Now().Add(3 * time.Minute) </ins> for i := 0; !passed && timeout.After(time.Now()); i++ { time.Sleep(2 * time.Second) Logf("About to make a proxy status call")

# See the License for the specific language governing permissions and # limitations under the License. <del> TAG = 1.7 </del> <ins> TAG = 1.8 </ins> PREFIX = gcr.io/google_containers all: push

"containers": [ { "name": "webserver", <del> "image": "gcr.io/google_containers/nettest:1.7", </del> <ins> "image": "gcr.io/google_containers/nettest:1.8", </ins> "imagePullPolicy": "Always", "args": [ "-service=nettest",

"containers": [ { "name": "webserver", <del> "image": "gcr.io/google_containers/nettest:1.7", </del> <ins> "image": "gcr.io/google_containers/nettest:1.8", </ins> "args": [ "-service=nettest", "-delay-shutdown=10"

"fmt" "io/ioutil" "log" <del> "math/rand" </del> "net/http" "os" "os/signal"

// Find all sibling pods in the service and post to their /write handler. func contactOthers(state *State) { <ins> const waitTimeout = 2 * time.Minute </ins> defer state.doneContactingPeers() client, err := client.NewInCluster() if err != nil {

log.Printf("Server version: %#vn", v) } <ins> for start := time.Now(); time.Since(start) < waitTimeout; time.Sleep(5 * time.Second) { eps := getWebserverEndpoints(client) if eps.Len() >= *peerCount { break } state.Logf("%v/%v has %v endpoints, which is less than %v as expected. Waiting for all endpoints to come up.", *namespace, *service, len(eps), *peerCount) } </ins> // Do this repeatedly, in case there's some propagation delay with getting // newly started pods into the endpoints list. for i := 0; i < 15; i++ { <del> endpoints, err := client.Endpoints(*namespace).Get(*service) if err != nil { state.Logf("Unable to read the endpoints for %v/%v: %v; will try again.", *namespace, *service, err) time.Sleep(time.Duration(1+rand.Intn(10)) * time.Second) } eps := sets.String{} for _, ss := range endpoints.Subsets { for _, a := range ss.Addresses { for _, p := range ss.Ports { eps.Insert(fmt.Sprintf("http://%s:%d", a.IP, p.Port)) } } } </del> <ins> eps := getWebserverEndpoints(client) </ins> for ep := range eps { state.Logf("Attempting to contact %s", ep) contactSingle(ep, state) } time.Sleep(5 * time.Second) } } <ins> //getWebserverEndpoints returns the webserver endpoints as a set of String, each in the format like "http://{ip}:{port}" func getWebserverEndpoints(client *client.Client) sets.String { endpoints, err := client.Endpoints(*namespace).Get(*service) eps := sets.String{} if err != nil { state.Logf("Unable to read the endpoints for %v/%v: %v.", *namespace, *service, err) return eps } for _, ss := range endpoints.Subsets { for _, a := range ss.Addresses { for _, p := range ss.Ports { eps.Insert(fmt.Sprintf("http://%s:%d", a.IP, p.Port)) } } } return eps } </ins> // contactSingle dials the address 'e' and tries to POST to its /write address. func contactSingle(e string, state *State) { body, err := json.Marshal(&WritePost{