Change:
// Find all sibling pods in the service and post to their /write handler. func contactOthers(state *State) { defer state.doneContactingPeers() <del> token, err := ioutil.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token") if err != nil { log.Fatalf("Unable to read service account token: %v", err) } cc := client.Config{ Host:        "https://" + net.JoinHostPort(os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT")), Version:     "v1beta3", BearerToken: string(token), Insecure:    true, // TOOD: package certs along with the token } client, err := client.New(&cc) if err != nil { log.Fatalf("Unable to create client:nconfig: %#vnerror: %vn", err) } if v, err := client.ServerVersion(); err != nil { log.Fatalf("Unable to get server version: %vn", err) } else { log.Printf("Server version: %#vn", v) </del> <ins> masterRO := url.URL{ Scheme: "http", Host:   os.Getenv("KUBERNETES_RO_SERVICE_HOST") + ":" + os.Getenv("KUBERNETES_RO_SERVICE_PORT"), Path:   "/api/" + latest.Version, </ins> } <ins> client := &client.Client{client.NewRESTClient(&masterRO, latest.Version, latest.Codec, false, 5, 10)} </ins> // Do this repeatedly, in case there's some propagation delay with getting // newly started pods into the endpoints list.

Discussion:
It's failing on both GCE and GKE-CI. It appears to have broken somewhere around build number 2509/2510 which includes a likely candidate PR (). /cc
I was about to debug this one. I'll take it.
Root cause is almost definitely one of these: Summary secrets example pod (details) the permissions on python files. (details) the implementation of DeepCopy to use reflection instead of Gob encode/decode. (details) networking test (details) back to executable. (details) protected docker processes (-900) (details) proxySocket out of to (details) Commit by Fix secrets example pod The file was modified Commit by Fixing the permissions on python files. The file was modified cluster/juju/charms/trusty/kubernetes- The file was modified cluster/juju/return-node- The file was modified cluster/juju/charms/trusty/kubernetes- Commit by Changing the implementation of DeepCopy to use reflection instead of Gob encode/decode. The file was The file was modified The file was modified Commit by Fix networking test The file was modified contrib/for-tests/network- The file was modified contrib/for-tests/network- The file was modified The file was modified contrib/for-tests/network-tester/Makefile Commit by Changing back to executable. The file was modified Commit by OOM protected docker processes (-900) The file was modified The file was modified The file was modified Commit by Move proxySocket out of to Moves proxySocket out of to new in proxy package in order to start separating proxy logic and implementation and make proxier more manageable to review. The file was modified The file was
Pretty sure it's commit , so rolling back now. cc .
The above rollback solved that immediate problem, but now the test, while proceeding healthily, is timing out. More debugging or optimization required: Networking should function for intra-pod communication ..... [irrelevant stuff deleted] ..... INFO: Proxy status call returned in 4.861944ms INFO: Attempt 35/60: test still running INFO: About to make a proxy status call Build timed out (after 90 minutes). Marking the build as aborted. Build timed out (after 90 minutes). Marking the build as failed. Ran 40 of 46 Specs in 5069.444 seconds FAIL!

-- 18 Passed 5 Skipped Build was aborted Recording test results
I'm not entirely sure why, but the test is running normally again, completing consistently in 13 seconds. Closing this issue for now. !