Change:
Analyzing the flake in every change will be queued until the minimum period has expired, The test works as follows: create a NodePort Service with one backend Pod check NodePort Service works make the backend Pod unready check that the NodePort Service doesn't work (kube-proxy should not forward traffic to the unready Pod) The test races because it tries to get to the NodePort before the kube-proxy programs the data-plane and remove the iptables-rules. Sometimes the test is faster than kube-proxy and the pod replies, the request succeeds , causing the test to fail At Nov 29 15:40:06.910: FAIL: expected error when trying to connect to NodePort address Expected an error to have occurred. the NodePort still works Kubelet The pod becomes unready at and the status is updated Controller-manager The endpoint slice controller process the pod update on the EndpointSlices at Kube/proxy here is the thing, kube-proxy queues the iptables update for 4 seconds and race with the test Kube-proxy 2 different proxies queue the call too, but at different times This explains flakes on this tests, since the pod is unready but still answer traffic ... I need to find since when and why are we using 10s as EDIT It seems that it was 10s for a long time in CI Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-soon
/sig testing The question is , why it started to be more evident after 1.25? I want to think that our testing framework is faster after moving to ginkgo v2, does it make sense?
this affects scalability tests IIUIC
I don't think that Ginkgo is involved much once this particular test runs: There are few calls, but that's it. What may have changed is the scheduling of concurrent tests, which may have an effect on the timing.
I was wondering if the code within a spec passes some kind of preprocessing that influences the time to execute each line of the test, I'm trying to make sense of this
I'll try to stare at this some more this morning. But one question - these are running in parallel with other tests, correct? Is it possible that a different container from a different test is coming up and getting assigned the same port? How do you all divvy up ports to ensure that kind of cross-test pollution never happens?
no no , sorry, the test is correct , sorry I'm misdirecting the issue, the thing I wanted to clarify is if there is some overhead when you have a test where the time to execute the or the time between and is affected
The handling of changed. This affects any log output produced by this test. I can't say whether it got faster, slower, or even is noticeable at all.
No - with the caveat that it's always possible that something weird/random is happening - nothing's changed that would cause a systematic slow down or speed up in that way. The By's aren't doing anything slow/fast - and you can comment them out for now if you want to rule it out but, truly, i doubt that's the source of any change.
just wanted to be sure, thanks for confirming
/remove-sig testing
Not sure why this would have started flaking more in 1.25, but anyway, there's a const for "the maximum time a kube-proxy daemon on a node is allowed to not notice a Service update, such as type=NodePort". It is currently 5 minutes, which the comment notes is way too large. But anyway, any test that depends on waiting for kube-proxy to process a Service/Endpoint change should poll for / waiting for it to happen.
yeah, but I don't want to carry with those hacks, we are in much better shape now regarding stability and I want to increase the bar so we detect subtles bugs ... we don't want test that just pass no matter what, we want test to give us confidence things work as expected ... if I have to create a service and wait for 5 minutes to start working we are doing something wrong
oh, I think that I'm doing a mistake, and you are spot on as always the test waits for the Service to not answer, and the connections to the ClusterIP fail, is the NodePort only the problem ... can we have some problem with the 1.25 improvements related to NodePorts only? nevermind, the ClusterIP check depends no the local kube-proxy, the NodePort depends on the kube-proxy of the respective Node, in this case the exec pod kube-proxy disabled the ClusterIP but the kube-proxy on the Node with the NodePort still didn't flushed the IPs ... kube-proxy's are not in sync flushing the rules

Discussion:
exit 1 fi fi <del> params+=" --iptables-sync-period=1m --iptables-min-sync-period=10s --ipvs-sync-period=1m --ipvs-min-sync-period=10s" </del> <ins> params+=" --iptables-sync-period=1m --iptables-min-sync-period=1s --ipvs-sync-period=1m --ipvs-min-sync-period=1s" </ins> if [[ -n "${KUBEPROXY_TEST_ARGS:-}" ]]; then params+=" ${KUBEPROXY_TEST_ARGS}" fi