Change:
Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).
The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions. It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back. I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.
Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.

Discussion:
// Only the addresses of the current family are returned. // IPv6 link-local and loopback addresses are excluded. GetLocalAddresses(dev string) (sets.Set[string], error) <ins> // GetAllLocalAddressesExcept return all local addresses on the node, except from the passed dev. // This is not the same as to take the diff between GetAllLocalAddresses and GetLocalAddresses // since an address can be assigned to many interfaces. This problem raised // https://github.com/kubernetes/kubernetes/issues/114815 GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) </ins> }

"net" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/klog/v2" </ins> utilproxy "k8s.io/kubernetes/pkg/proxy/util" netutils "k8s.io/utils/net"

} return true } <ins> // GetAllLocalAddressesExcept return all local addresses on the node, // except from the passed dev.  This is not the same as to take the // diff between GetAllLocalAddresses and GetLocalAddresses since an // address can be assigned to many interfaces. This problem raised // https://github.com/kubernetes/kubernetes/issues/114815 func (h *netlinkHandle) GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) { ifaces, err := net.Interfaces() if err != nil { return nil, err } var addr []net.Addr for _, iface := range ifaces { if iface.Name == dev { continue } ifadr, err := iface.Addrs() if err != nil { // This may happen if the interface was deleted. Ignore // but log the error. klog.ErrorS(err, "Reading addresses", "interface", iface.Name) continue } addr = append(addr, ifadr...) } return utilproxy.AddressSet(h.isValidForSet, addr), nil } </ins>

return nil, fmt.Errorf("netlink is not supported in this platform") } <ins> // GetAllLocalAddressesExcept is part of interface. func (h *netlinkHandle) GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) { return nil, fmt.Errorf("netlink is not supported in this platform") } </ins> // Must match the one in proxier_test.go func (h *netlinkHandle) isValidForSet(ip net.IP) bool { return false

klog.ErrorS(err, "Error listing addresses binded to dummy interface") } // nodeAddressSet All addresses *except* those on the dummy interface <del> nodeAddressSet, err := proxier.netlinkHandle.GetAllLocalAddresses() </del> <ins> nodeAddressSet, err := proxier.netlinkHandle.GetAllLocalAddressesExcept(defaultDummyDevice) </ins> if err != nil { klog.ErrorS(err, "Error listing node addresses") } <del> nodeAddressSet = nodeAddressSet.Difference(alreadyBoundAddrs) </del> hasNodePort := false for _, svc := range proxier.svcPortMap {

if proxier.ipvsScheduler == "mh" { serv.Flags |= utilipvs.FlagSourceHash } <del> if err := proxier.syncService(svcPortNameString, serv, true, alreadyBoundAddrs); err == nil { </del> <ins> // We must not add the address to the dummy device if it exist on another interface shouldBind := !nodeAddressSet.Has(serv.Address.String()) if err := proxier.syncService(svcPortNameString, serv, shouldBind, alreadyBoundAddrs); err == nil { </ins> activeIPVSServices.Insert(serv.String()) <del> activeBindAddrs.Insert(serv.Address.String()) </del> <ins> if shouldBind { activeBindAddrs.Insert(serv.Address.String()) } </ins> if err := proxier.syncEndpoint(svcPortName, svcInfo.ExternalPolicyLocal(), serv); err != nil { klog.ErrorS(err, "Failed to sync endpoint for service", "servicePortName", svcPortName, "virtualServer", serv) }

return res, nil } <ins> func (h *FakeNetlinkHandle) GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) { res := sets.New[string]() for linkName := range h.localAddresses { if linkName == dev { continue } for _, addr := range h.localAddresses[linkName] { if h.isValidForSet(addr) { res.Insert(addr) } } } return res, nil } </ins> // SetLocalAddresses set IP addresses to the given interface device.  It's not part of interface. func (h *FakeNetlinkHandle) SetLocalAddresses(dev string, ips ...string) error { if h.localAddresses == nil {

"k8s.io/kubernetes/pkg/proxy/ipvs" ) <ins> // (I am unsure if this test has any value since it only tests the fake implementation) </ins> func TestSetGetLocalAddresses(t *testing.T) { fake := NewFakeNetlinkHandle(false) _ = ipvs.NetLinkHandle(fake) // Ensure that the interface is honored

if !addr.Equal(expected) { t.Errorf("Unexpected mismatch, expected: %v, got: %v", expected, addr) } <del> fake.SetLocalAddresses("kube-ipvs0", "4.3.2.1") </del> <ins> fake.SetLocalAddresses("kube-ipvs0", "1.2.3.4", "4.3.2.1") </ins> addr, _ = fake.GetAllLocalAddresses() expected = sets.New("1.2.3.4", "4.3.2.1") if !addr.Equal(expected) { t.Errorf("Unexpected mismatch, expected: %v, got: %v", expected, addr) } <ins> addr, _ = fake.GetAllLocalAddressesExcept("kube-ipvs0") expected = sets.New("1.2.3.4") if !addr.Equal(expected) { t.Errorf("Unexpected mismatch, expected: %v, got: %v", expected, addr) } </ins> }