Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:

Discussion:
// inter-pod affinity for pods of different rollouts and anti-affinity // for pods of the same rollout, so it will need to be updated when // performing a rollout. <del> setAffinity(d) </del> <ins> setAffinities(d, false) </ins> d.Spec.Strategy.RollingUpdate = &appsv1.RollingUpdateDeployment{ MaxSurge:       intOrStrP(1), MaxUnavailable: intOrStrP(0),

framework.Logf("Updating label deployment %q pod spec (iteration #%d)", name, i) deployment, err = e2edeploy.UpdateDeploymentWithRetries(c, ns, d.Name, func(update *appsv1.Deployment) { update.Spec.Template.Labels["iteration"] = fmt.Sprintf("%d", i) <del> setAffinity(update) </del> <ins> setAffinities(update, true) </ins> }) framework.ExpectNoError(err)

} } <del> func setAffinity(d *appsv1.Deployment) { d.Spec.Template.Spec.Affinity = &v1.Affinity{ PodAffinity: &v1.PodAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []v1.WeightedPodAffinityTerm{ </del> <ins> // setAffinities set PodAntiAffinity across pods from the same generation // of Deployment and if, explicitly requested, also affinity with pods // from other generations. // It is required to make those "Required" so that in large clusters where // scheduler may not score all nodes if a lot of them are feasible, the // test will also have a chance to pass. func setAffinities(d *appsv1.Deployment, setAffinity bool) { affinity := &v1.Affinity{ PodAntiAffinity: &v1.PodAntiAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []v1.PodAffinityTerm{ </ins> { <del> Weight: int32(100), PodAffinityTerm: v1.PodAffinityTerm{ TopologyKey: "kubernetes.io/hostname", LabelSelector: &metav1.LabelSelector{ MatchExpressions: []metav1.LabelSelectorRequirement{ { Key:      "name", Operator: metav1.LabelSelectorOpIn, Values:   []string{d.Spec.Template.Labels["name"]}, }, { Key:      "iteration", Operator: metav1.LabelSelectorOpNotIn, Values:   []string{d.Spec.Template.Labels["iteration"]}, }, </del> <ins> TopologyKey: "kubernetes.io/hostname", LabelSelector: &metav1.LabelSelector{ MatchExpressions: []metav1.LabelSelectorRequirement{ { Key:      "name", Operator: metav1.LabelSelectorOpIn, Values:   []string{d.Spec.Template.Labels["name"]}, }, { Key:      "iteration", Operator: metav1.LabelSelectorOpIn, Values:   []string{d.Spec.Template.Labels["iteration"]}, </ins> }, }, }, }, }, }, <del> PodAntiAffinity: &v1.PodAntiAffinity{ </del> <ins> } if setAffinity { affinity.PodAffinity = &v1.PodAffinity{ </ins> RequiredDuringSchedulingIgnoredDuringExecution: []v1.PodAffinityTerm{ { TopologyKey: "kubernetes.io/hostname",

}, { Key:      "iteration", <del> Operator: metav1.LabelSelectorOpIn, </del> <ins> Operator: metav1.LabelSelectorOpNotIn, </ins> Values:   []string{d.Spec.Template.Labels["iteration"]}, }, }, }, }, }, <del> }, </del> <ins> } </ins> } <ins> d.Spec.Template.Spec.Affinity = affinity </ins> }