Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I tried ipvs mode in kube-proxy instead of iptables. It works fine with all service types except loadbalancer. I am using MetalLb as loadbalancer and I can not make it work. In iptables mode it works perfect, I just do not want go back to it from ipvs Environment: Kubernetes version (use ):MetalLb
I've just tested on a kubeadm 1.9.3 cluster, with IPVS mode enabled on kube-proxy. Both and seem to be ignored by kube-proxy in IPVS mode, so external traffic is completely unrouteable. In contrast, kube-proxy in iptables mode creates DNAT/SNAT rules for external and loadbalancer IPs. This should be a blocker for IPVS mode GA, because it will break load-balancer implementations in all major clouds, as well as various bare-metal implementations like and custom solutions relying on .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
More labels and signal boosting... /area ipvs should this be listed as a blocker in ?
Updated to reflect this bug. This is interesting because we recently a new e2e test job which runs the ingress tests in a cluster with ipvs (https://k8s-). The tests have been steadily green... Note: those tests are not run using kubeadm. To clarify, what do you mean by "ignore"?
Okay, did some more testing to verify my claims. Here's the situation in my cluster. There's a single node, which is both master and worker (removed the master taint). On it, I've installed Calico as the network addon, and MetalLB to provide the plumbing (setting LoadBalancer IPs, and responding to ARP queries about the IP). I defined a single LoadBalancer service, pointing to an nginx pod running on the same machine. MetalLB allocated an IP from my LAN: As you can see, there's an external IP, set through the LoadBalancer status field: From the perspective of a client on the same machine, I can't connect. times out trying to connect. On this client, the ARP cache shows that MetalLB did its job, and told the client to send packets to the k8s machine: On the k8s machine, shows that the SYN packets arrive at the machine, but no response ever comes back: Looking at , I see that kube-proxy did program a balancer: Okay... What could be eating the packets... Looking at the firewall... Aha! The DROP counter is increasing with every SYN packet. So, Calico isn't processing IPVS-forwarded packets correctly. And indeed, looking at the release notes for Calico 3.0.1 (I'm running 2.6) : So, it looks like Calico was relying on kube-proxy iptables behavior, and needed to be updated to support IPVS mode. I'm going to update my version of Calico, and check if that resolves the problem.
Scratch that, it still doesn't work. To be certain, I switched from Calico to Flannel, to simplify the networking setup. I see the same behavior as above, IPVS is configured, but the packets don't match any rules in the iptables FORWARD table, and get dropped. If I switch back to iptables mode in kube-proxy, the forwarding works as expected.
Contents of my FORWARD table, with flannel installed. The DROPed packets from policy are the SYN packets for my service. Another datapoint: this is a Debian Testing machine. I can't think of anything debian-specific that would cause this, but it might be the reason why your e2e tests aren't failing?
Okay, bisecting stuff... Looks like it's Docker CE 17.12 that sets the default policy to DROP on the FORWARD table. Now I'm wondering if this is all just a compatibility problem with Docker 17.12 (k8s only officially supports 17.03 currently). I'm trying to find an old .deb of docker 17.03 to test this theory.
Nope, no luck, I downgraded to Docker 17.03, and things fail exactly the same way - no matching rules in FORWARD, so the service packets just get dropped. Again, this only happens in IPVS mode, in iptables mode the service functions correctly. Based on this, I still have no idea why e2e tests aren't failing, assuming you're testing against the latest 1.9.3... FYI, to enable IPVS mode, I edited the kube-proxy configmap in the running cluster, the SupportIPVSProxyMode=true feature gate, and changed the mode to "ipvs". Is there anything else I need to do?
Also, looking at the test grid, I see https://k8s- . That suite is failing "[sig-network] Services should be able to up and down services". I wonder if that's related?...
Even more testing: I also tried with , to check if the problem is specific to . also doesn't work, same failure mode: packets arrive at machine, and iptables FORWARD drops them.
Another datapoint (sorry for the spam, I'm recording this as I discover new things). I tried setting , just to test the theory, and now I get an ~immediate "No route to host" when trying to talk to my service. So, the FORWARD thing might just be a red herring, afaict IPVS is just not processing the packets at all! To confirm: shows that now, when the machine receives the SYN, it sends a bunch of ARP who-has to try and find the owner of the service IP. This fails (because this IP is virtual), so after a few retries, the machine sends back an ICMP Unreachable packet to signal forwarding failure. Digging some more...
Aha! I got it working! So, there are two separate problems with the 1.9.3 IPVS implementation: -proxy is not adding IPs from and to the interface. is a missing iptables rule in the FORWARD table, to allow the external traffic to transit the node. If I manually fix both of these things (add the service IP to kube-ipvs0, and switch the FORWARD table policy to ACCEPT), the service works perfectly. The first issue seems like an easy fix, it's just a couple more IPs to sync to kube-ipvs0. The second one... I'm not sure, obviously we can't just authorize all traffic forwarding, so it has to be a targeted rule for each service... But I don't know if the rule applies pre- or post-IPVS, so I don't know what the packet match should look like. And it might reintroduce some scaling issues that IPVS was meant to fix, I don't know. OTOH, if this rule is only needed for external services, that's way fewer services than all the cluster-internal ones!
One more note before I head to bed: it looks like the e2e tests you pointed at are using L7 Ingress on GCP. AFAIK, the implementation of GCP Ingress does not use LoadBalancer or externalIPs at all, instead it forwards traffic to the NodePort on the machine. I need to check this tomorrow, but if I'm right, that means those e2e tests are not at all exercising the behavior of LoadBalancer services and externalIPs.
I promise this is my last update for tonight :) I a rule in iptables to log the packets that are getting dropped, with . It turns out, it's the packets from the pod back to the external client that are getting dropped! The SYN packet is correctly getting routed to my serving pod, but the SYN/ACK response doesn't match any forwarding rules, and gets dropped. I don't understand (yet) what happens in kube-proxy iptables mode that makes this traffic flow work, but breaks in IPVS mode. But that's a mystery for tomorrow, unless someone beats me to it :)
you can not just add loadbalancer IP to kube-ipvs0. In that case each node in the cluster will respond to ARP requests. But only and only metallb should do this. If we add IP to kube-ipvs0 we must somehow prevent linux from responding to ARP requests on that IP....
Looping in
Good point, thanks for catching that! The easiest "fix" for that is to define a second kube-ipvs1 interface, set the NOARP flag on that interface, and add LoadBalancer and externalIPs to that interface. That way, the machine still "owns" the IPs, but doesn't try to respond to ARP queries. I don't know if adding another interface has other side-effects on IPVS. I'm going to spend time today tracing packets in IPVS and iptables mode, to try and better understand the firewalling decisions for the 2nd issue.
a dup of
I am on vocation now and will be back soon
That looks very much like the FORWARD drops, yes! Thanks for the pointer, I will test-drive your proposed fix and my proposed fix for the IPVS interface thing, as manual edits to my network stack. If those two edits work, I think the rest is just a matter of PRs to upstream the changes into kube-proxy. I'm happy to help with those changes, if someone who knows the codebase can point me in the right general direction :)
Confirmed, the forwarding rules suggested as the fix for also fixes external IPs and LoadBalancer IPs. Specifically, the rule that fixes them is , which permits the SYN/ACK and subsequent packets to transit from the pod to the physical network. Note that both fixes are needed: the externalIP/LoadBalancer IP must be to an interface on the machine, or IPVS does not handle the inbound traffic. For this I used kube-ipvs1, which is a copy of kube-ipvs0 with ARP handling disabled. the firewall fix is needed to allow the pod to respond to cluster-external clients. This is the shell script I used to "manually" implement this fix. It's quite hacky, but it illustrates the things that kube-proxy needs to do. You can run this on a k8s node to make an externalIP or LoadBalancer IP work in IPVS mode. If you would like me to contribute PRs to any of this, please let me know - I'm happy to help out, but I don't want to step on anyone's toes if someone else already has something :)
I am working on fixing
Ack, I'll leave that to you then, I believe it will also fix the LoadBalancer use case. About kube-ipvs1... Does that seem like a good solution to the problem (machine must have service IP defined, but kernel must not respond to ARP) ? If so, should I send a PR for it, or does someone else want to work on it?
You may need to be a bit careful so that not breaking nodeport in IPVS, xref: .
This rule is missing (11.0.0.0/16 is my own addr of course); iptables -t nat -A KUBE-SERVICES ! -s 11.0.0.0/16 -m set  --match-set KUBE-EXTERNAL-IP dst,dst -j KUBE-MARK-MASQ The corresponding for KUBE-CLUSTER-IP is there already; iptables -t nat -A KUBE-SERVICES ! -s 11.0.0.0/16 -m set  --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ I am not using loadBalancer but the problem seem the same with just an externalIP in the service. With adding the externalIP to kube-ipvs0 and the iptables rule above it works fine.
That's strange, just KUBE-MARK-MASQ should not be enough to fix the forwarding issue, from my testing... There seems to be explicitly rules missing. As for adding an IP to kube-ipvs0 - as stated above, that works but has the unintended side-effect that the machine might respond to ARP queries for the IP, if the external IP belongs to a local network. Using a separate kube-ipvs* interface fixes that.
The forwarding may be a loadBalancer thing. I just set an externalIP on a (ClusterIP) service. The kube-ipvs0 is DOWN and NOARP on my system; It seems like it's only purpose is to direct packets to ipvs by hosting a set of addresses. It doesn't even have to be up it seems(?).
For there are 2 bugs in this area; If "externalIPs" is used, those addresses must be to kube-ipvs0 otherwise no traffic to those addresses is sent to ipvs. The proposed kube-ipvs1 is identical with the existing kube-ipvs0, both are NOARP and dummy interfaces. kube-ipvs0 is safer because it is left in DOWN state. I can't se any reason for creating kube-ipvs1. Second is that the set is defined, filled with addresses but not used in any iptables rule. It shall be used in exactly the same way as the KUBE-CLUSTER-IP set; I am unsure if I shall write a new issue for externalIPs and ipvs. The address setting on kube-ipvs0 is clearly the same as in this issue, but I don't know about the set.
In
The iptables rule is there in 1.9.3 (and onwards) The externalIPs are not to kube-ipvs0 even in 1.11. If the parameter mentioned in the comment above is set to everything works.
The externalIPs are still not to kube-ipvs0 in v1.10.0
If we set the NodeIP as the externalIP, it should work.
Should we add an e2e test case for service?
I saw that GCE is used for e2e tests, but I have read that GCE with "type: LoadBalancer" does LB directly to the endpoints (pods). That reduces one redundant LB-step (the kube-proxy) but makes kube-proxy tests in GCE useless. I might be wrong about the direct LB so please verify it, I will try to find where I read it.
Figured out one reason I thought GCE bypasses the kube-proxy; GCE when . But kube-proxy always masquerades, ;
I think it is necessary, I didn't found any related case in our e2e test. Sorry, but I'm not quiet understand your concern, I think specific user case would be better. : )
Please forget my GCE/kube-proxy comments I think I got it wrong. Sorry.
I tried to patch false-true in and re-built 1.10.0. It makes externalIPs work but I get open listen-port in main netns for the container ports; My external address is 10.0.0.1 and it is assigned to which is (!). And I get conflicts if the port is used in main netns (e.g open with address); The problem does not be so simple to fix as I hoped. The opening of ports in main netns seems to be inherited from the handling. The nodeports are opened in main netns by to verify that they are free to use and to prevent that the port is used by others once it is assigned as a . For this should be avoided. In the same way as for and some care must be assumed from the user. In my example I opened port 23 (telnet) in the pod and the node had this port open for address in main netns. This is perfectly ok and also works if it wasn't for this main netns check.
To be able to use in I set a local route to the externalIP on all nodes, e.g; By using (which is always there if the dummy module is loaded) the kube-proxy does not "sync" it away. If the address is manually to it will be removed after a short while. With this work-around and works for me.
Will this issue be fixed by the fix for as well?
I guess so. I am going to close this issue in favor of discreating issues. Please feel free to re-open or create a new issue if the bug still exists. /close
Does now get IPs for and ? I don't see any changes for that, and it's a requirement to make things work in addition to the iptables rules changes.
I think always get IPs for and , according to , if we create an ipvs real server rule, we will definitely bind this IP to .
It doesn't, unfortunately. I just tested this by loading a v1.11 test build of kube-proxy into my bare metal cluster, switched kube-proxy to IPVS mode, and the service IPs are not programmed on kube-ipvs0, only ClusterIP services. shows that there are rules for loadBalancer and externalIPs, but the IPs are no present in , so IPVS never handles the traffic. This is with image gs://kubernetes-release-dev/ci/v1.11.0-alpha.0.2614+-bazel/bin/linux/amd64/kube- , which afaict is the very latest dev build of kube-proxy.
Data from my cluster: As you can see, I have a loadBalancer IP programmed. IPVS has a rule for 192.168.16.240...
And just to check: if I add the service IP manually to kube-ipvs0, the service starts working. So, it looks like did fix the second issue in this report, which was that iptables dropped the traffic - but the other issue with missing IPs on the IPVS interface is still happening.
Specifically: and are incorrect, must be for externalIPs and loadBalancer IPs as well, not just for ClusterIP.
Okay, I will send another fix ASAP. THANKS!
If you add the address manually it will be "synced away" after a minute or so. For a longer lasting work-around, please see my comment above.
Yeah, I was just verifying that everything except the addr bind was working correctly. Fortunately, in my case, I can just disable IPVS mode to get back to a working cluster, until v1.11. I just want to make sure that it all works in 1.11 ;)
An please consider my command about the ports opened by in main netns. If you just alter false-true in the lines above the will open for instance port 80 for the external address in main netns, but this will fail if a http server is bound to 0.0.0.0:80 in main netns! The bind is there to protect the NodePorts but MUST NOT be there for externalIPs.
Looking at the code, I don't understand how setting bindAddr=true in the syncService calls triggers a listening port in kube-proxy. Looking at the code, bindAddr only triggers a netlink sync to add the IP to the dummy interface, no other behavior... Are we talking about the same false-true, or a different piece of the code? Specifically, in the lines I linked above, the only effect of changing false-true is that gets called, which does the equivalent of . There's no logic in there that triggers any port bindings.
I don't know where it is done, just that it is. When altering the code in the same way you did I actually got exactly the port-80 conflict described above.
To check, use in main netns.
I altered the code in 1.10.0 not 1.11. That may make a difference though...
If externalIP happen to be Node-Local IP, then kube-proxy will try to HOLD the port. Please check the code: I think it's an expected behavior and we did the same thing in iptables proxier. I want to clarify it before starting the code developing.
Ok, I see. It should not be hard to just avoid binding to port in main netns on the nodes. But is there a reason for this local bind? I see the reason for NodePort but not in this case.
Seems like the fix is in. Can this be closed?
yeahlet us close it /close
Is there a workaround for k8s v1.9.5? Thanks.
--proxy-mode=iptables works fine. And perhaps you can use if ipvs is necessary for you. Kube-router replaces kube-proxy and has more options than kube-proxy in ipvs mode, for instance direct server returm (DSR).
Sorry, Can you elaborate more on how/where to set the --proxy-mode=iptables? I'm running on the AWS platform and trying to fix the type=LoadBalancer external ip from <pending.
We suggest you to check out to v1.11.
I have switched to kube-router and removed kube-proxy completely. Kube-router gives you IPVS proxy and routing in a single software
BTW setting ARP OFF on kube dummy interface is actually not enough. System begins reply on ARP requests with MAC of any another available interface. To solve this problem an additional action is required This means reply only if IP matches interface IP. Default value 0 make system reply on if IP matches any IP on the system including kube dummy interface
what if just use:
Hi, bug is still there. Kube-proxy in IPVS mode (and Kube-router too) adds IP-addresses to dummy interface . The problem is that ARP-replies still going out even if dummy-interface have NOARP flag set. The funy fact that for small clusters it is not so problematic, you can even be so glad that you don't need any external loadbalancer for external IPs, but it is wrong behavior and it is should be fixed. The similar bug report for The solution: It was described here:
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
could you reopen, please?
cc Can you take a look? Is there any side effect for doing this? I understand that is's a global system variable and we are changing the default values.
/reopen I agree with We should investigate the potential side effect before making that change.
Reopening this issue. <detailsIn response to : /reopen I agree with We should investigate the potential side effect before making that change. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Well, this change just enables the next check for all network interfaces: if the arp packet come to the interface which have specific IP, then send arp-answer. if interface have no this IP-address (it may be assigned to another interface), then ignore. Basically it's shouldn't brake anything. But there is maybe few applications which uses this hack, eg. romana-vip may have issues after this change.
As an option we can add some warning about this: Or even fallback to iptables mode. But as I said before, some users find this bug so useful, so they may be unhappy of that :)
I think we can do the sysctl setting now since many IPVS users complain about the issue. We can keep watching the test grid and community issue report to see if will cause other problems. Can anyone raise a PR to fix that? I can merge it soon.
I'll do that

Discussion:
const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" const sysctlForward = "net/ipv4/ip_forward" <ins> const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce" </ins> // Proxier is an ipvs based proxy for connections between a localhost:lport // and services that provide the actual backends.

} } <ins> // Set the arp_ignore sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpIgnore); val != 1 { if err := sysctl.SetSysctl(sysctlArpIgnore, 1); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpIgnore, err) } } // Set the arp_announce sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpAnnounce); val != 2 { if err := sysctl.SetSysctl(sysctlArpAnnounce, 2); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpAnnounce, err) } } </ins> // Generate the masquerade mark to use for SNAT rules. masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue)