Change:
! it seems all the job is timing out 05-04 ,%20ipvs,%20IPv6,%20master No response No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area ipvs
possible pre-submit or infra related issue? I saw this event/warning in most of the cases.
where do you see that error? The jobs are timing out and started to fail at one point in time, the ipvs ipv4 jobs are not failing, usually that indicates there is some trigger, that may be a change in kubernetes/test-infra or in kubernetes/kubernetes, as first check we should verify the changes that went in between the last good and the first bad run
In the events tab of Job Pod Info section
They're configured to run in the same host cluster so the same things should be available. ci-kubernetes-kind-network-ipvs-dual and ci-kubernetes-kind-network-ipvs-ipv6
the following test times out for IPVS-IPv6 on local kind cluster after 1 hr. The same test for IPv4 passes in 11secs.
I stand corrected then, why does it times out? are there some very large timeouts?
looking into it
you can run that test alone with verbosity using and see in which part is blocked
will probably fix the job timeouts
Logs for the failing job just before it crashed.
for reference , talking with Daman offline it seems that the tests that curl inside a pod timeout because the request hangs forever. It only happens on IPVS with IPv6
We talked about this on the SIG Network call today, suggested we assign you /assign Let us know if you need help moving it forward! :vulcan_salute:
Is there any reason why we pass nil instead of healthzServer while instantiating the IPVS IPv6 proxier? <brThis results in a nil pointer exception on health check node port servers and the read lock is never released: <brThe write lock in the SyncServices call will never be acquired since the read lock is not released <brthis will ultimately block the reconciliation loop, and kube-proxy hangs.
it should not, commented in the PR ok, in golang the http server recovers from panics on handlers, that explains why the whole proxy does not goes down. that second unlock is wrong, it has to be done before calling can you. please submit a fix for that too?
yeah, the proxy goes down because the HTTP handler panicked before releasing the read lock. Sure, will submit a fix for that too.
the job is working fine now - ,%20ipvs,%20IPv6,%20master&show-stale-tests= All greens expect [It] [sig-network] Services should fail health check node port if there are only terminating endpoints This test case will work after is merged.
Thanks Daman, you did a great job narrowing this down and fixing it
Yes, thanks for the quick turnaround it's very appreciated!! :vulcan_salute:

Discussion:
exec, syncPeriod, minSyncPeriod, filterCIDRs(true, excludeCIDRs), strictARP, tcpTimeout, tcpFinTimeout, udpTimeout, masqueradeAll, masqueradeBit, localDetectors[1], hostname, nodeIP[1], <del> nil, nil, scheduler, ipFamilyMap[v1.IPv6Protocol], kernelHandler) </del> <ins> recorder, healthzServer, scheduler, ipFamilyMap[v1.IPv6Protocol], kernelHandler) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv6 proxier: %v", err) }