Change:
Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready. However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:

Discussion:
serviceMap   proxy.ServiceMap endpointsMap proxy.EndpointsMap nodeLabels   map[string]string <ins> // initialSync is a bool indicating if the proxier is syncing for the first time. // It is set to true when a new proxier is initialized and then set to false on all // future syncs. // This lets us run specific logic that's required only during proxy startup. // For eg: it enables us to update weights of existing destinations only on startup // saving us the cost of querying and updating real servers during every sync. initialSync bool </ins> // endpointSlicesSynced, and servicesSynced are set to true when // corresponding objects are synced after startup. This is used to avoid updating // ipvs rules with some partial data after kube-proxy restart.

serviceChanges:        proxy.NewServiceChangeTracker(newServiceInfo, ipFamily, recorder, nil), endpointsMap:          make(proxy.EndpointsMap), endpointsChanges:      proxy.NewEndpointChangeTracker(hostname, nil, ipFamily, recorder, nil), <ins> initialSync:           true, </ins> syncPeriod:            syncPeriod, minSyncPeriod:         minSyncPeriod, excludeCIDRs:          parsedExcludeCIDRs,

return } <ins> // its safe to set initialSync to false as it acts as a flag for startup actions // and the mutex is held. defer func() { proxier.initialSync = false }() </ins> // Keep track of how long syncs take. start := time.Now() defer func() {

} if curEndpoints.Has(ep) { <ins> // if we are syncing for the first time, loop through all current destinations and // reset their weight. if proxier.initialSync { for _, dest := range curDests { if dest.Weight != newDest.Weight { err = proxier.ipvs.UpdateRealServer(appliedVirtualServer, newDest) if err != nil { klog.ErrorS(err, "Failed to update destination", "newDest", newDest) continue } } } } </ins> // check if newEndpoint is in gracefulDelete list, if true, delete this ep immediately uniqueRS := GetUniqueRSName(vs, newDest) if !proxier.gracefuldeleteManager.InTerminationList(uniqueRS) {

continue } } // Delete old endpoints for _, ep := range curEndpoints.Difference(newEndpoints).UnsortedList() { // if curEndpoint is in gracefulDelete, skip

proxier.servicesSynced = true } <ins> func makeEndpointSliceMap(proxier *Proxier, allEpSlices ...*discovery.EndpointSlice) { for i := range allEpSlices { proxier.OnEndpointSliceAdd(allEpSlices[i]) } proxier.mu.Lock() defer proxier.mu.Unlock() proxier.endpointSlicesSynced = true } </ins> func makeTestService(namespace, name string, svcFunc func(*v1.Service)) *v1.Service { svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{

} } <ins> func Test_syncEndpoint_updateWeightsOnRestart(t *testing.T) { tcpProtocol := v1.ProtocolTCP ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, v1.IPv4Protocol) svc1 := makeTestService("ns1", "svc1", func(svc *v1.Service) { svc.Spec.ClusterIP = "10.20.30.41" svc.Spec.Ports = []v1.ServicePort{{ Name:     "p80", Port:     int32(80), Protocol: v1.ProtocolTCP, }} }) epSlice1 := makeTestEndpointSlice("ns1", "svc1", 1, func(eps *discovery.EndpointSlice) { eps.AddressType = discovery.AddressTypeIPv4 eps.Endpoints = []discovery.Endpoint{{ Addresses: []string{"10.180.0.1"}, }} eps.Ports = []discovery.EndpointPort{{ Name:     pointer.StringPtr("p80"), Port:     pointer.Int32(80), Protocol: &tcpProtocol, }} }) // sync proxy rules to get to the desired initial state makeServiceMap(fp, svc1) makeEndpointSliceMap(fp, epSlice1) fp.syncProxyRules() serv := &utilipvs.VirtualServer{ Address:   netutils.ParseIPSloppy("10.20.30.41"), Port:      uint16(80), Protocol:  string(tcpProtocol), Scheduler: fp.ipvsScheduler, } vs, err := fp.ipvs.GetVirtualServer(serv) if err != nil { t.Errorf("failed to get virtual server, err: %v", err) } rss, err := fp.ipvs.GetRealServers(vs) if err != nil { t.Errorf("failed to get real servers, err: %v", err) } for _, rs := range rss { rs.Weight = 0 if err = fp.ipvs.UpdateRealServer(vs, rs); err != nil { t.Errorf("failed to update real server: %v, err: %v", rs, err) } } // simulate a restart by enabling initial sync logic. fp.initialSync = true err = fp.syncEndpoint(proxy.ServicePortName{ NamespacedName: types.NamespacedName{ Name:      "svc1", Namespace: "ns1", }, Port:     "80", Protocol: tcpProtocol, }, true, vs) if err != nil { t.Errorf("failed to sync endpoint, err: %v", err) } rss, err = fp.ipvs.GetRealServers(vs) if err != nil { t.Errorf("failed to get real server, err: %v", err) } for _, rs := range rss { if rs.Weight != 1 { t.Logf("unexpected realserver weight: %d, expected weight: 1", rs.Weight) t.Errorf("unexpected realserver state") } } } </ins> func TestIPv4Proxier(t *testing.T) { tcpProtocol := v1.ProtocolTCP tests := []struct {