Change:
What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that. But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets

Discussion:
fs.DurationVar(&o.config.IPVS.SyncPeriod.Duration, "ipvs-sync-period", o.config.IPVS.SyncPeriod.Duration, "The maximum interval of how often ipvs rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.") fs.DurationVar(&o.config.IPVS.MinSyncPeriod.Duration, "ipvs-min-sync-period", o.config.IPVS.MinSyncPeriod.Duration, "The minimum interval of how often the ipvs rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').") fs.StringSliceVar(&o.config.IPVS.ExcludeCIDRs, "ipvs-exclude-cidrs", o.config.IPVS.ExcludeCIDRs, "A comma-separated list of CIDR's which the ipvs proxier should not touch when cleaning up IPVS rules.") <ins> fs.BoolVar(&o.config.IPVS.StrictARP, "ipvs-strict-arp", o.config.IPVS.StrictARP, "Enable strict ARP by setting arp_ignore to 1 and arp_announce to 2") </ins> fs.DurationVar(&o.config.ConfigSyncPeriod.Duration, "config-sync-period", o.config.ConfigSyncPeriod.Duration, "How often configuration from the apiserver is refreshed.  Must be greater than 0.") fs.BoolVar(&o.config.IPTables.MasqueradeAll, "masquerade-all", o.config.IPTables.MasqueradeAll, "If using the pure iptables proxy, SNAT all traffic sent via Service cluster IPs (this not commonly needed)") fs.StringVar(&o.config.ClusterCIDR, "cluster-cidr", o.config.ClusterCIDR, "The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead")

config.IPVS.SyncPeriod.Duration, config.IPVS.MinSyncPeriod.Duration, config.IPVS.ExcludeCIDRs, <ins> config.IPVS.StrictARP, </ins> config.IPTables.MasqueradeAll, int(*config.IPTables.MasqueradeBit), config.ClusterCIDR,

// excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch // when cleaning up ipvs services. ExcludeCIDRs []string <ins> // strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries // from kube-ipvs0 interface StrictARP bool </ins> } // KubeProxyConntrackConfiguration contains conntrack settings for

out.MinSyncPeriod = in.MinSyncPeriod out.Scheduler = in.Scheduler out.ExcludeCIDRs = *(*[]string)(unsafe.Pointer(&in.ExcludeCIDRs)) <ins> out.StrictARP = in.StrictARP </ins> return nil }

syncPeriod    time.Duration minSyncPeriod time.Duration // Values are CIDR's to exclude when cleaning up IPVS rules. <del> excludeCIDRs   []string </del> <ins> excludeCIDRs []string // Set to true to set sysctls arp_ignore and arp_announce strictARP      bool </ins> iptables       utiliptables.Interface ipvs           utilipvs.Interface ipset          utilipset.Interface

syncPeriod time.Duration, minSyncPeriod time.Duration, excludeCIDRs []string, <ins> strictARP bool, </ins> masqueradeAll bool, masqueradeBit int, clusterCIDR string,

} } <del> // Set the arp_ignore sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpIgnore); val != 1 { if err := sysctl.SetSysctl(sysctlArpIgnore, 1); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpIgnore, err) </del> <ins> if strictARP { // Set the arp_ignore sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpIgnore); val != 1 { if err := sysctl.SetSysctl(sysctlArpIgnore, 1); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpIgnore, err) } </ins> } <del> } // Set the arp_announce sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpAnnounce); val != 2 { if err := sysctl.SetSysctl(sysctlArpAnnounce, 2); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpAnnounce, err) </del> <ins> // Set the arp_announce sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpAnnounce); val != 2 { if err := sysctl.SetSysctl(sysctlArpAnnounce, 2); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpAnnounce, err) } </ins> } }

ipvs:              ipvs, ipset:             ipset, clusterCIDR:       "10.0.0.0/24", <ins> strictARP:         false, </ins> hostname:          testHostname, portsMap:          make(map[utilproxy.LocalPort]utilproxy.Closeable), portMapper:        &fakePortOpener{[]*utilproxy.LocalPort{}},

// excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch // when cleaning up ipvs services. ExcludeCIDRs []string `json:"excludeCIDRs"` <ins> // strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries // from kube-ipvs0 interface StrictARP bool `json:"strictARP"` </ins> } // KubeProxyConntrackConfiguration contains conntrack settings for