Code change:
cc
Reopening to remember to check if helped
No flakes in recent history
This just failed on PR Jenkins for me:
I've been hitting this one repeatedly in https://doc-0l3o4-0hqt8-s-
Access Denied for that link
Jenkins has already purged the link. Anyone who sees this - please open issues WITH THE ERROR MESSAGE if this pops up again. A Jenkins link is limited-lifetime.
this just re-occurred, here's the log:
Can you take a look, with prio, please. Of note:
I just randomly started debugging this because it caused a PR I was reviewing to fail e2es. I think the problem is the test is unaware that kube-proxies can fall behind each other. This is very likely because the "relist timeout" imposed on the watch is randomized server side, also one node could be low on cpu etc. Take a single failing case: To break down what happened: Created webserver-f0m07, tested it was up Picked some node (104.197.253.149) and tested reachability Scheduled hostexec pod on some node (e2e-gce-master-3-minion-w6ut, 104.197.244.14) Ran the ss command and checked for nodeport Evident from the following console logs: 09:05:09 Jan 27 09:04:45.111: INFO: Waiting up to 5m0s for pod webserver-f0m07 status to be running 09:05:09 Jan 27 09:04:45.119: INFO: Found pod 'webserver-f0m07' on node 'e2e-gce-master-3-minion-bcr0' 09:05:09 STEP: trying to dial each unique pod 09:05:09 Jan 27 09:04:45.278: INFO: Controller webserver: Got non-empty result from replica 1 [webserver-f0m07]: 1 of 1 required successes so far 09:05:09 STEP: hitting the pod through the service's NodePort 09:05:09 Jan 27 09:04:45.354: INFO: Successfully reached http://104.197.253.149: 09:05:09 STEP: verifying the node port is locked 09:05:09 Jan 27 09:04:45.376: INFO: Waiting up to 5m0s for pod hostexec status to be running 09:05:09 Jan 27 09:04:57.437: INFO: Found pod 'hostexec' on node 'e2e-gce-master-3-minion-w6ut' 09:05:09 Jan 27 09:04:57.441: INFO: Running '/jenkins-master-data/jobs/kubernetes-pull-build-test-e2e-gce/workspace --server=https://104.197.246.4 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-pull-build-test-e2e- exec --namespace=e2e-tests-services-oht70 hostexec -- /bin/sh -c for i in $(seq 1 10); do if ss -ant46 'sport = :' | grep ^LISTEN; then exit 0; fi; sleep 0.1; done; exit 1' 09:05:09 STEP: deleting service nodeportservice-test in namespace e2e-tests-services-oht70 09:05:09 STEP: stopping RC webserver in namespace e2e-tests-services-oht70 09:05:09 [AfterEach] Services node the webserver pod landed on: node the hostexec pod landed on: Comparing kube-proxy logs from these 2 nodes we see that the second kubeproxy (hostexec) only actually processed the OnUpdate event for the node port at 17:05:17, which is actually waay later than the first kubeproxy sees it (17:04:40). To rule out clock skew I check when the kubelet on the first node saw the hostexec, and this matches up to roughly the time the kubelet on the second node saw the webserver pod (~17:45). To figure out where the skew comes from just check time between OnEndpointsUpdate log lines, you'll see the hostexec kubeproxy taking 2s at times (~17:04:29), while the webserver kube-proxy just whizzes along at 0.5s per update. So I think the problems here are: 's no way to know that all kube-proxies have executed the nodeport actions for a service (i.e "service readiness") -proxy maybe processing OnUpdates in-efficiently, but that's not the core problem. test should either have a much larger timeout (the hostexec sh command above has seq(1 10)) and/or make sure it lands on the right host it has already verified preconditions for. The "service readiness" problem is a little deeper and will probably help ingress too.
chatted with Prashanth. He is investigating this.
And again today: gs://kubernetes-jenkins/logs/kubernetes-e2e-gce-parallel/ In the last 20 days this has failed ~9 times in that suite, only counting runs where only a few tests failed. 0.8% fail rate.
this is now assigned to you - plans for a fix?
for review
PR for fix is in.

Related discussion:
text: ip := pickNodeIP(c) testReachable(ip, nodePort) <ins> By("verifying the node port is locked") </ins> hostExec := LaunchHostExecPod(f.Client, f.Namespace.Name, "hostexec") <del> cmd := fmt.Sprintf(`ss -ant46 'sport = :%d' | tail -n +2 | grep LISTEN`, nodePort) </del> <ins> // Loop a bit because we see transient flakes. cmd := fmt.Sprintf(`for i in $(seq 1 10); do if ss -ant46 'sport = :%d' | grep ^LISTEN; then exit 0; fi; sleep 0.1; done; exit 1`, nodePort) </ins> stdout, err := RunHostCmd(hostExec.Namespace, hostExec.Name, cmd) if err != nil { Failf("expected node port (%d) to be in use, stdout: %v", nodePort, stdout)