Change:
Which jobs are failing: pull-kubernetes-e2e-gce Which test(s) are failing: [sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] Since when has it been failing: At least since 9/30 (). But that's almost at the end of what's currently visible on the testgrid, so maybe longer Testgrid link: Not sure what I'm supposed to put here. ? Reason for failure: The test case connects to a service every 2 seconds for 1 minute and checks if it hit the same endpoint at least 15 times in a row. But the individual checks are taking unexpectedly long and so sometimes the test doesn't even manage to run 15 times total in 1 minute. eg: Oct 8 13:32:01.402: INFO: Running ' --server=https://34.82.143.14 -- exec --namespace=services-1060 execpod-affinityklpfn -- /bin/sh -x -c curl -q -s --connect-timeout 2 http://10.40.0.5:/' Oct 8 13:32:07.627: INFO: stderr: "+ curl -q -s --connect-timeout 2 http://10.40.0.5:/n" Oct 8 13:32:07.627: INFO: stdout: "affinity-nodeport-2z9hv" Oct 8 13:32:07.627: INFO: Received response from host: affinity-nodeport-2z9hv Oct 8 13:32:09.402: INFO: Running ' --server=https://34.82.143.14 -- exec --namespace=services-1060 execpod-affinityklpfn -- /bin/sh -x -c curl -q -s --connect-timeout 2 http://10.40.0.5:/' Oct 8 13:32:16.027: INFO: stderr: "+ curl -q -s --connect-timeout 2 http://10.40.0.5:/n" Oct 8 13:32:16.027: INFO: stdout: "affinity-nodeport-2z9hv" Oct 8 13:32:16.027: INFO: Received response from host: affinity-nodeport-2z9hv Note the gap between the "Running" line and the "stderr" line. It's not clear if the slowness is in connecting to the execpod or in connecting from the execpod to the affinity service. It is also not particularly consistent between attempts. (Here it was 6s, 1s, 2s, 3s, 5s, 7s, 7s, 3s, 4s, 6s, 7s, 8s.) Anything else we need to know: It looks the checks are also slow in successful runs, it's just that they're not quite as slow, so they pass. I'm not sure if the problem is "something is making the cluster slow and so this totally valid test is taking too long and failing" or "e2e-gce has grown over time and now cluster load has reached the point where this formerly-passing test now has unreasonable performance expectations". /sig network

Discussion:
func CheckAffinity(execPod *v1.Pod, serviceIP string, servicePort int, shouldHold bool) bool { serviceIPPort := net.JoinHostPort(serviceIP, strconv.Itoa(servicePort)) cmd := fmt.Sprintf(`curl -q -s --connect-timeout 2 http://%s/`, serviceIPPort) <del> timeout := TestTimeout </del> <ins> timeout := AffinityTimeout </ins> if execPod == nil { timeout = LoadBalancerPollTimeout }

// TestTimeout is used for most polling/waiting activities TestTimeout = 60 * time.Second <ins> // AffinityTimeout is the maximum time that CheckAffinity is allowed to take; this // needs to be more than long enough for AffinityConfirmCount HTTP requests to // complete in a busy CI cluster, but shouldn't be too long since we will end up // waiting the entire time in the tests where affinity is not expected. AffinityTimeout = 2 * time.Minute </ins> // AffinityConfirmCount is the number of needed continuous requests to confirm that // affinity is enabled. AffinityConfirmCount = 15