Change:
AKS had a customer report repeated issues in their clusters where: -proxy would redeploy (e.g. due to AKS deploying a new kube-proxy image with CVE fixes) c-ares DNS client would send repeated DNS queries to the kube-dns service VIP from the same src IP address, creating a UDP conntrack entry to the svc VIP and keeping it alive. For example, with kube-dns svc VIP 192.168.0.10, conntrack showed: queries to the svc VIP were blackholed until the customer manually ran a command like: Issue occurred only in clusters with many services and endpoints (~10k services and ~6k endpoints). However, the customer has seen this issue repeatedly for months, about 1-5 times per week across their clusters. kube-proxy code to delete stale UDP conntrack entries should have deleted the conntrack entry to kube-dns svc VIP automatically in the first sync after service and endpointslice cache is initialized. Unfortunately, the customer said they were unable to reproduce this issue in other environments outside of their production clusters. AKS engineers were also unable to repro this issue. Update (2024-08-03): I'm now able to reproduce the issue using a DNS client that reuses the same src IP / port and clears DNAT conntrack entries between queries, in a k8s 1.29.7 cluster with 2,000 services. The bug is triggered reliably in this setup with The test client code, scripts, and steps to reproduce are in this repository: The customer shared kube-proxy logs (at verbosity "3") from when the issue occurred. Unfortunately, I don't have permission to share the full logs publicly, but I think what I'm seeing in the logs gives a clue about what's happening. 05:00:02. nodes, endpointslices, and services all synced: -dns endpointslices processed after , but before iptables sync: iptables sync at 05:00:02. shows 4658 services, which is much less than the number of services in the cluster. (We can't see if kube-dns is one of these services, since kube-proxy at doesn't log which services were processed.) at 05:00:02. and the next sync at 05:00:05., we see thousands of lines that look like services and endpointslices updates: sync at 05:00:05. shows 9605 services: I believe this could explain why kube-proxy does not clean up the stale UDP conntrack entry: kube-proxy detects that the endpointslice and service caches are synced using respectively. Comments on the say that this indicates that the informer cache is up-to-date, but warns "Note that this doesn't tell you if an individual handler is synced!! For that, please call HasSynced on the handle returned by AddEventHandler." This comment, along with the output of kube-proxy logs, makes me suspect that kube-proxy is initiating a sync before it has received and processed all the services and endpoints in the informer cache. If so, this could explain why the "delete conntrack" codepath is skipped. kube-proxy deletes stale conntrack entries for UDP services that are updated from and are in . If in the first sync is missing kube-dns, but includes kube-dns, then kube-proxy would . Subsequent syncs would see at least one endpoint for kube-dns, so would continue to skip deletion. If this theory is correct, then I wonder if kube-proxy could change the initialization check to ensure that all pre-sync events from the informer cache are delivered before the first iptables sync, maybe like this: Should kube-proxy wait until all pre-sync events are delivered before setting ? kube-proxy version is v1.28.5 Note that the kube-proxy image that AKS built for 1.28.5 includes the backported patch to fix this race condition in iptables partial sync: Azure Kubernetes Service Linux (I don't have the exact version but could get it if it's relevant) N/A N/A N/A
Possibly this issue would be addressed by
/sig network
you can workaround this in the meantime with an init container flushing all udp entries `
Just to clarify, an init container on the kube-proxy pod?
If I'm reading the code correctly, when kube-dns is missing from , then the iptables restore on the first sync won't install DNAT rules for kube-dns If so, I think even with an init container flushing conntrack, there's still a period between iptables restore and the next iptables sync where a DNS client could create a conntrack entry to the svc VIP.
I see, I don't have time to test it, but then you need to do two things up the kube-proxy iptables rules to avoid to have stale entries to services with endpoints that does not exist the existing UDP entries directed to Services a strawman approach to add to the existing kube-proxy manifest
joined us on the call today, and confirmed that we should move forward with the patch he proposed: /triage accepted /assign
I'm now able to reproduce the bug reliably using the test client and steps here: After applying the patch to wait for , the issue no longer occurs.

Discussion:
// NewEndpointSliceConfig creates a new EndpointSliceConfig. func NewEndpointSliceConfig(ctx context.Context, endpointSliceInformer discoveryv1informers.EndpointSliceInformer, resyncPeriod time.Duration) *EndpointSliceConfig { result := &EndpointSliceConfig{ <del> listerSynced: endpointSliceInformer.Informer().HasSynced, logger:       klog.FromContext(ctx), </del> <ins> logger: klog.FromContext(ctx), </ins> } <del> _, _ = endpointSliceInformer.Informer().AddEventHandlerWithResyncPeriod( </del> <ins> handlerRegistration, _ := endpointSliceInformer.Informer().AddEventHandlerWithResyncPeriod( </ins> cache.ResourceEventHandlerFuncs{ AddFunc:    result.handleAddEndpointSlice, UpdateFunc: result.handleUpdateEndpointSlice,

// NewServiceConfig creates a new ServiceConfig. func NewServiceConfig(ctx context.Context, serviceInformer v1informers.ServiceInformer, resyncPeriod time.Duration) *ServiceConfig { result := &ServiceConfig{ <del> listerSynced: serviceInformer.Informer().HasSynced, logger:       klog.FromContext(ctx), </del> <ins> logger: klog.FromContext(ctx), </ins> } <del> _, _ = serviceInformer.Informer().AddEventHandlerWithResyncPeriod( </del> <ins> handlerRegistration, _ := serviceInformer.Informer().AddEventHandlerWithResyncPeriod( </ins> cache.ResourceEventHandlerFuncs{ AddFunc:    result.handleAddService, UpdateFunc: result.handleUpdateService,

resyncPeriod, ) <ins> result.listerSynced = handlerRegistration.HasSynced </ins> return result }