Change:
We are running self managed Kubernetes clusters on GCP and use Services of with both public and private s to create load balancers for our clusters. Since upgrading kube-proxy to 1.26.3 (from 1.25.5) we saw that all healthchecks from GCP started to fail and traffic is now blindly load balanced across all nodes, which broke Services with set to . The issue seems to be introduced with this change: Here, ipset is introduced and is used in iptables on chain as: seems to include external IPs (as also noted here: and here: ) so the last rule above is dropping packets to external IPs that are not routed before reaching this point. The next chain in iptables is and contains a single rule: which seems specific to route packets arriving to health check ports Workarounds to get health checks to work again from GCP are to either flush ipset (temporary since kube-proxy will fight back) or move the ule to route packets to health check packets on top of : Health checks from GCP should continue to work as intended after upgrading kube-proxy from 1.25.5 to 1.26.3 Use kube-proxy v1.26.3 in a Kubernetes cluster on GCP and create a Service of type . Health checks from GCP should fail on all nodes Rolling back to v1.25.5 seems to fix the issue, even though we have to roll to fresh nodes to clean up everything and make health checks work again. <details</details<detailsGCP </details<details</detailsNo response <detailscontainerd://1.6.8 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/cc /area ipvs
/assign
I can't recreate this problem on K8s v1.26.4. I use a simple service: And get: The wget is from an external node on the same segment. 192.168.1.2 is the address to a node where a server exist, 192.168.1.4 to a node without a server. Is this happening in combination with I.e. are you using a node-ip as loadBalancerIP?
Hi I have the same problem. A packet capture reveals, in GCP the healthcheck is directed to nodes with the LB IP as destination ip. So in order to create this problem either create a static arp entry for the loadBalancer IP, and wget http://loadBalancerIP: or spinup a pod in the cluster and do a http://loadBalancerIP:
Yeah, this is related to my comment here , the loadbalancer IP is as a local route to the host
Thanks, I have treated them as separate issues. I will check again.
yes, as pointed out, in order to reproduce this you need to make requests to the LB address. I can also curl kube-proxy on localhost fine, but gcp goes via the LB addresses to health check
To use http://loadbalancerip:/ doesn't make sense to me unless you have a one-node cluster with loadbalancerip==nodeip. The probe is to check if an endpoint is present on a particular node, hence the loadbalancerip can't really be used. But the comment: indicates that you have many nodes. I don't get it.
It appears that GCE ILBs rely on Direct Server Return (DSR) as their load balancing mechanism. DSR only modifies the destination MAC address and is utilized during health checks across all VMs within the associated instance groups. This is likely due to K8S being just one of many use cases, and health checks may be conducted on the same port as the load-balanced service. In such cases, the destination IP serves as the distinguishing factor on the host.
yes, I have many nodes on my clusters. Catching health check traffic on a node for the following service as an example: gives: The controller configures a health check with 8 seconds interval on port , which is also visible from the packets arriving on the node. I am not 100% on how GCP routes for those health checks but feels like comment above is spot on.
So to summarise; GCP sends probes to "loadbalancerip:health-check-port" in intervals and explicitly set the mac address to target different nodes. Not the cleanest solution imho, and very hard to imagine beforehand. Here is what I suggest, add a rule before the reject in KUBE-IPVS-IPS:
yes, this is a workaround that seems to do the trick (I also mention that one in the issue description). Currently, kube-proxy already adds such a rule, but this happens in chain which is evaluated after . I think it is a matter of ordering rules
I don't want to reorder, because some other corner-case might pop up. But to add that rule, and keep everything else as-is feels safer. It will however expose the healthCheckPort externally via the loadBalancerIP. In GCP I am sure that is prevented by some fw-rule somewhere, but other K8s installation may be affected.
Many thanks for your explanations
I have created a , but I can't test that it really solves the problem on GCP. Can any of you please apply the PR and verify that it works on GCP?
Would it be possible to configure GCP to use the same port as the load-balanced service also for K8s? I.e. don't use the healthCheckPort at all. A problem may be that a request to a node without a server may be black-holed, i.e. no reject either icmp or http.
this is more complex since is how cloud-providers loadbalancers implemented it, using externalTraficPolicy: local, the health check is on the , for externalTraficPolicy: cluster, some cloud-providers use the applications port and others use the kube-proxy healtcheck, that it runs on any of the node ips
Imo key things are: and: So from GCP perspective they are not aware of what's running on the cluster. K8s or something else. So the probing for available servers are generic, "K8S being just one of many use cases". In that case I assumed that the cluster admin may configure how probing is done. But I don't know GCP so it may not be so, but... On the other hand if GCP is aware that this is actually a K8s system, the "K8S being just one of many use cases" doesn't hold and GCP should probe the node-ip's instead of using some MAC-hack.
I can confirm that I have compiled the PR, updated the kube-proxy ds image, and tested it on GCP. It is working as expected! :fireworks: You're welcome! I'm glad I could be of help. Your Metallb patches have also been very useful to me in the past. Thank you for fixing this issue! this is more complex since is how cloud-providers loadbalancers implemented it Although I could patch the cloud-provider-gcp controller for my own use, this wouldn't solve the issue for other users or other cloud controllers. Additionally, using the service port itself may result in hitting the application directly, which may not always be desired.

Discussion:
"-m", "set", "--match-set", proxier.ipsetList[kubeExternalIPSet].Name, "dst,dst", "-j", "RETURN") proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), <ins> "-m", "set", "--match-set", proxier.ipsetList[kubeHealthCheckNodePortSet].Name, "dst", "-j", "RETURN") proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), </ins> "-m", "conntrack", "--ctstate", "NEW", "-m", "set", "--match-set", proxier.ipsetList[kubeIPVSSet].Name, "dst", "-j", "REJECT")