Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --UDP traffic to a Pod behind a LoadBalancer Service exposing a UDP listener service is blackholed when the replacement Pod is scheduled on the same node as the "old" Pod. A new Pod that gets provisioned (replacing the previous one) will receive traffic once it's ready. a UDP Listener Pod (single pod) and using nodeSelector ensure it runs on a specific node the Pod behind a LoadBalancer Service continuous stream of UDP packets from outside the cluster to the External IP of the Service that traffic is correctly received the Pod and observe as a new one is created, traffic should now be blackholed and not reaching the "new" Pod After the new Pod is recreated, we noticed the following entries on the node: notice that the <SVC EXT IPappears here both in the original and the NATed , this is wrong and I suspect happens because the conntrack for the old Pod were flushed before the iptables rules for the new Pod were created (?) and in the interim new packets have reached the node and resulted in these incorrect conntrack entries. To "fix" this, we manually deleted the above conntrack entries, and this allowed new correct one to be established based on the now existing iptables nat rule: with this new conntrack entries, the traffic started flowing as expected to the new Pod and notice that the NATed now correctly shows the Pod IP (172.30.3.219) instead of the Service External IP. While spelunking through the GitHub issues and PRs, I found this issue which was addressed in , it does mention that: the service has endpoints available. Since the fix in this PR refers to NodePort, perhaps the same also applies to External IP (i.e. LoadBalancer Service?) or there's something else at play here? AFAICT, this fix is included in v1.20.7 that we are running. Kubernetes version (use ):Cloud provider or hardware configuration: GCP/GKE OS (e.g: ):Kernel (e.g. ): Install tools: Terraform (with google provider) Network plugin and version (if this is a network-related bug): Others:
/sig network
According to GCP support (since we hit this on a GKE cluster) there's already a fix for this bug and it in v1.21, however I'm unable to find to link it here (hope they can provide the link shortly).
GCP support referenced as the fix for this issue, however this was already merged and released in v1.20.4 but we are running v1.20.7, so should have the fix already, so the status is unclear at this point.
thanks for the detailed report, at first sight we do clean the external ips is it possible to set kube-proxy in a log level of 2 and capture the logs from the node to correlate with the time it happends, and understand why those entries are not being deleted?
Hey we can try that, however since this is a GKE environment I'd have to check how/if we can change kube-proxy log level. My understanding of the issue is that it's not that the conntrack entries are flushed, is that they are flushed before the nat iptables rules are created, causing faulty conntrack entries to be created in the interim (since the udp packets are incoming all the time).
I'm having some miscommunication about the availability of fix in with GCP/GKE support and was hoping you could confirm that is included in v1.20.5 (as per the release notes there? ) and per the cherry-pick PR GCP Support claim that this change is incorporated in v1.21 and we should wait for v1.21 to be available in GKE.
I see the commit in 1.20.5 , this is the cherry-pick The conntrack entries are always deleted after the conntrack entries are created, there is some code that detects that a service goes stale and based on that it deletes the entries. I think that current code detects this case, at least I know it works when the service goes from 0 to 1 endpoint, that is why the logs will be useful, we can see exactly what happens ...
Thanks for confirming I just checked and seem to be already running in log level 2: I looked back at the logs when this issue was identified and I see the following entries: This two messages and only these two repeat every time we tried to rollout the deployment. is indeed the and is the ClusterIP of the service: but every time we tried, we ended up with these stale conntrack entries I mentioned in the OP and only after deleting them would the udp packet start flowing into the "new" pod.
oh wait, by external IP you mean ?
/triage accepted /assign
here you go, having good reports on bugs make them easy to spot ;)
indeed! I apologize if I wasn't clear indeed I was referring to the LoadBalancer Service External IP. Thanks for fix!!! But now I'm a bit confused, since I understood that LoadBalancer Service External IP == LoadBalancer Ingress IP, but I see these are two separate attributes? so for example: These are the same IP, but internally they are separate attributes?
hmm I don't know how kubectl prints the information, but you can check the fields with and you could see that ExternalIPs and LoadBalancer.IngressIP are different fields, actually one is in the Status and other in the Spec
interesting, so in the output, I can only see the , there's actually no mention of here: That's why I always assumed that this is referring to the same thing/attribute/field.

Discussion:
for _, extIP := range svcInfo.ExternalIPStrings() { conntrackCleanupServiceIPs.Insert(extIP) } <ins> for _, lbIP := range svcInfo.LoadBalancerIPStrings() { conntrackCleanupServiceIPs.Insert(lbIP) } </ins> nodePort := svcInfo.NodePort() if svcInfo.Protocol() == v1.ProtocolUDP && nodePort != 0 { klog.V(2).Infof("Stale %s service NodePort %v -> %d", strings.ToLower(string(svcInfo.Protocol())), svcPortName, nodePort)

// Delete ClusterIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) <ins> // Delete ExternalIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) // Delete LoadBalancerIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) </ins> // Delete NodePort entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc)

fp.exec = &fexec svcIP := "10.20.30.41" <ins> extIP := "1.1.1.1" lbIngressIP := "2.2.2.2" </ins> svcPort := 80 nodePort := 31201 svcPortName := proxy.ServicePortName{

makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *v1.Service) { svc.Spec.ClusterIP = svcIP <ins> svc.Spec.ExternalIPs = []string{extIP} svc.Spec.Type = "LoadBalancer" </ins> svc.Spec.Ports = []v1.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: v1.ProtocolUDP, NodePort: int32(nodePort), }} <ins> svc.Status.LoadBalancer.Ingress = []v1.LoadBalancerIngress{{ IP: lbIngressIP, }} </ins> }), )

) fp.syncProxyRules() <del> if fexec.CommandCalls != 2 { t.Fatalf("Updated UDP service with new endpoints must clear UDP entries") </del> <ins> if fexec.CommandCalls != 4 { t.Fatalf("Updated UDP service with new endpoints must clear UDP entries 4 times: ClusterIP, NodePort, ExternalIP and LB") </ins> } <del> // Delete ClusterIP Conntrack entries expectCommand := fmt.Sprintf("conntrack -D --orig-dst %s -p %s", svcIP, strings.ToLower(string((v1.ProtocolUDP)))) actualCommand := strings.Join(fcmd.CombinedOutputLog[0], " ") if actualCommand != expectCommand { t.Errorf("Expected command: %s, but executed %s", expectCommand, actualCommand) </del> <ins> // the order is not guaranteed so we have to compare the strings in any order expectedCommands := []string{ // Delete ClusterIP Conntrack entries fmt.Sprintf("conntrack -D --orig-dst %s -p %s", svcIP, strings.ToLower(string((v1.ProtocolUDP)))), // Delete ExternalIP Conntrack entries fmt.Sprintf("conntrack -D --orig-dst %s -p %s", extIP, strings.ToLower(string((v1.ProtocolUDP)))), // Delete LoadBalancerIP Conntrack entries fmt.Sprintf("conntrack -D --orig-dst %s -p %s", lbIngressIP, strings.ToLower(string((v1.ProtocolUDP)))), // Delete NodePort Conntrack entrie fmt.Sprintf("conntrack -D -p %s --dport %d", strings.ToLower(string((v1.ProtocolUDP))), nodePort), </ins> } <del> // Delete NodePort Conntrack entrie expectCommand = fmt.Sprintf("conntrack -D -p %s --dport %d", strings.ToLower(string((v1.ProtocolUDP))), nodePort) actualCommand = strings.Join(fcmd.CombinedOutputLog[1], " ") if actualCommand != expectCommand { t.Errorf("Expected command: %s, but executed %s", expectCommand, actualCommand) </del> <ins> actualCommands := []string{ strings.Join(fcmd.CombinedOutputLog[0], " "), strings.Join(fcmd.CombinedOutputLog[1], " "), strings.Join(fcmd.CombinedOutputLog[2], " "), strings.Join(fcmd.CombinedOutputLog[3], " "), } sort.Strings(expectedCommands) sort.Strings(actualCommands) if !reflect.DeepEqual(expectedCommands, actualCommands) { t.Errorf("Expected commands: %v, but executed %v", expectedCommands, actualCommands) </ins> } }