Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-master-slow Which test(s) are failing: [sig-network] Services should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] Since when has it been failing: This test started to be flaky on 11/18 Testgrid link: https://k8s- Reason for failure: Looks to be a pretty straightfowards test not passing: If we look at triage, this test has always been flaky, but it got significantly more flaky (2X failures) on 10/27: Anything else we need to know: This job is non-blocking for 1.13, except if this represents an actual bug in Affinity the 1.13 team needs to know ASAP. This failure may or may not be related to . It looks like this test was disabled for large clusters, but maybe the problem wasn't the cluster size. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig network /priority important-soon attn:
FWIW this seems to be a long standing issue, atleast as far bck as 1.12 timeframe
Sure. But the failures have become 3X as frequent, so I still want a sign-off by sig-network.
cc
Ack. Will work on this.
we plan to lift code freeze for 1.13 tomorrow,11/28 5pm PST. If there is a new regression in 1.13 thats increased the flakes and needs addressing we need to get the fix in asap. Can you please let us know if this is a blocker for 1.13 by tomorrow, 11/28, morning 10am PST ?
Great observation. I checked on the commits around that date but wasn't able to find any change that may cause regression. Hasn't been able to reproduce the same failure locally as well.
I just noticed that the same test is flaking a lot more on 1.12 branches as well https://k8s-
Good catch on that, so the recent increase of flakiness likely is not caused by 1.13 changes so we shouldn't block release on this either. cc
But we will be looking deeper into the root cause.
Several observations: It only happens when is false. When it's true, the test does more stuff before actually check service session affinity. It doesn't happen when executing in pod, probably due to the same reason above, the test start test pod before actually testing service. I wrote a script which does the same thing in my cluster of version 1.11.3-gke.18 in us-central1-a. cannot reproduce it. The hypothesis is that the service session affinity spec takes some time to reconcile after the service is created. It could happen either on the node or on the GCLB. Since I haven't been able to reproduce it's hard to tell. But it's more likely on GCLB side, even it could be regional issue. Looks like a sleep of seconds will deflake it. but it's just hiding issues. I would rather open another issue to track the real issue.
The most recent failure is here: Basically what happened is GCLB doesn't honor session affinity and still sends the request to different VMs. The log shows that 3 requests are routed to 3 different VMs. Dec 19 07:40:58.094: INFO: [service-fsn65 service-bf8nc] Dec 19 07:40:58.094: INFO: Affinity should hold but didn't. Dec 19 07:40:58.196: INFO: [pod,node] pairs: [{Pod:service-bf8nc Node:bootstrap-e2e-minion-group-f3nb} {Pod:service-fsn65 Node:bootstrap-e2e-minion-group-9sp7} {Pod:service-rnrcw Node:bootstrap-e2e-minion-group-m5rj}]; err: <nil
More recent failures: Logs:
Hello, could this issue be prioritized to help with stabilizing the release blocking upgrade tests? They're currently flaking ~23% of the time in () ~30% of the time in ()
/assign can you take a look
GitHub didn't allow me to assign the following users: take, a, look, can, you. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign can you take a look Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am preparing a PR, basically allow Affinity some time to be configured, instead of failing out instantly.
Thanks! Will keep an eye out for it.
The test is no longer flaky: https://k8s- Thanks everyone!
Thanks! Will keep an eye on it for a few days, on the following dashboards:
It's stabler now, but still flaking once/twice a day, e.g. in . - Is this the same issue as you looked at? Is there more to be done here? /reopen
Reopened this issue. <detailsIn response to : It's stabler now, but still flaking once/twice a day, e.g. in . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we should probably cherrypick into 1.13 in case of deflaking upgrade/downgrade job, because those jobs would run 1.13 tests against 1.14 (upgraded) cluster or 1.13 (downgraded) cluster. FYI we've hit similar situation for DNS test (ref ).
Good point. Cherrypick PR prepared.
14: <img width="1110" alt="Screen Shot 2019-03-16 at 12 11 07 PM" src="https://user-"13: <img width="1091" alt="Screen Shot 2019-03-16 at 12 11 22 PM" src="https://user-"Do we want to cherry-pick this back further?
Thanks for the nice graph. Look like the fix works. I'll create cherrypick PRs for 1.11 and 1.12 right away.
AFAICT this has been significantly improved: ! Closing this and thanks to /close
Closing this issue. <detailsIn response to : AFAICT this has been significantly improved: ! /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
// Record the response going to a given host. func (at *affinityTracker) recordHost(host string) { at.hostTrace = append(at.hostTrace, host) <ins> Logf("Received response from host: %s", host) </ins> } // Check that we got a constant count requests going to the same host.

} // CheckAffinity function tests whether the service affinity works as expected. <del> // If affinity is expected and transitionState is true, the test will // return true once affinityConfirmCount number of same response observed in a // row. If affinity is not expected, the test will keep observe until different // responses observed. The function will return false only when no expected // responses observed before timeout. If transitionState is false, the test will // fail once different host is given if shouldHold is true. func CheckAffinity(jig *ServiceTestJig, execPod *v1.Pod, targetIp string, targetPort int, shouldHold, transitionState bool) bool { </del> <ins> // If affinity is expected, the test will return true once affinityConfirmCount // number of same response observed in a row. If affinity is not expected, the // test will keep observe until different responses observed. The function will // return false only in case of unexpected errors. func CheckAffinity(jig *ServiceTestJig, execPod *v1.Pod, targetIp string, targetPort int, shouldHold bool) bool { </ins> targetIpPort := net.JoinHostPort(targetIp, strconv.Itoa(targetPort)) cmd := fmt.Sprintf(`wget -qO- http://%s/ -T 2`, targetIpPort) timeout := ServiceTestTimeout

if !shouldHold && !affinityHolds { return true, nil } <del> if shouldHold { if !transitionState && !affinityHolds { return true, fmt.Errorf("Affinity should hold but didn't.") } if trackerFulfilled && affinityHolds { return true, nil } </del> <ins> if shouldHold && trackerFulfilled && affinityHolds { return true, nil </ins> } return false, nil }); pollErr != nil {

Expect(err).NotTo(HaveOccurred(), "failed to fetch pod: %s in namespace: %s", execPodName, ns) if !isTransitionTest { <del> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true, false)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true)).To(BeTrue()) </ins> } if isTransitionTest { svc = jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.SessionAffinity = v1.ServiceAffinityNone }) <del> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, false, true)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, false)).To(BeTrue()) </ins> svc = jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.SessionAffinity = v1.ServiceAffinityClientIP }) <del> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true, true)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, execPod, svcIp, servicePort, true)).To(BeTrue()) </ins> } }

port := int(svc.Spec.Ports[0].Port) if !isTransitionTest { <del> Expect(framework.CheckAffinity(jig, nil, ingressIP, port, true, false)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, nil, ingressIP, port, true)).To(BeTrue()) </ins> } if isTransitionTest { svc = jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.SessionAffinity = v1.ServiceAffinityNone }) <del> Expect(framework.CheckAffinity(jig, nil, ingressIP, port, false, true)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, nil, ingressIP, port, false)).To(BeTrue()) </ins> svc = jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.SessionAffinity = v1.ServiceAffinityClientIP }) <del> Expect(framework.CheckAffinity(jig, nil, ingressIP, port, true, true)).To(BeTrue()) </del> <ins> Expect(framework.CheckAffinity(jig, nil, ingressIP, port, true)).To(BeTrue()) </ins> } }