Change:
continuing discussion from and
/sig network /triage accepted
from I think we'll officially declare that we support kernel 5.4+ and nftables command-line 0.9.2+, and we can use a expression to check that. (Which means really we'll support kernel 5.3, but we won't advertise it that way.) FTR (and maybe worth documenting in a comment wherever we put the check): If we decide we need concatenations-with-ranges support again later, that would imply 5.6/0.9.4. If we want to use in sets/maps, that would require at least 0.9.4 (and maybe later depending on exactly how we're using it; we'd have to re-check the release notes). Support for comments on tables/chains/sets/maps/etc (anything but rules and elements) didn't come until 5.10/0.9.7, but knftables itself already does a for that at construct time, and just drops those comments if they're not supported.
There was a and about nft segfaults. It appears that specifically, nft v0.9.4 to v0.9.8 (inclusive) will segfault if asked to do anything when any chain in any table contains a certain kind of set/map as created by nft v1.0.3 or later. Specifically, rules like these trigger it (I think specifically it's caused by using and together, which the anonymous map as-generated-by-1.0.3+ uses. Using a named map rather than an anonymous one does not help.) Note that with nft < 1.0.1, doing any operation requires parsing the entire current ruleset. So that means when kube-proxy is using nft 1.0.3 or newer, nothing else in the host network namespace can use nft 0.9.4 - 0.9.8 for anything_. if the system firewall or something else in the host network namespace is using nft 1.0.3 or newer, then it might create rules that will break kube-proxy if kube-proxy is using nft 0.9.4 - 0.9.8. If kube-proxy is using nft 1.0.1 or later, then it should not be possible for any other well-behaved component to break it, even if there are future bugs like the current one, because kube-proxy never looks at any tables but its own, so it wouldn't see things that other components did that it couldn't understand. OK, so what does this mean for us? We could just ship the official kube-proxy images with hand-built nft 0.9.2 rather than whatever's current in Debian. Then we guarantee we don't install rules other people can't deal with. Except that then we potentially run into problems if the system uses a much newer nft with some new bug that breaks 0.9.2. (We could do something -esque here, but ... ugh.) I think we actually want to go the opposite way, and have kube-proxy print a warning at startup if your nft is older then 1.0.1, because older versions run the risk of being broken by something someone else is doing in another table. But then there's the problem of us breaking other people. Or, even if we don't break other software, we still make it that admins on some systems would be unable to debug nftables-based problems... I guess we could avoid using the known-problematic syntax... the simplest way to do this would be to revert back to doing endpoints the way that iptables does: of course, that's less efficient, but...
Sounds to me that 1.0.2 is the golden version. It doesn't break 0.9.8, and it doesn't parse the entire current ruleset.
golden relative to the bugs we know about right now anyway... but maybe building our official images with 1.0.2 for a while would be a good plan... (Unfortunately we would have to actually build it ourselves. We can't just pull a 1.0.2 build from a random older Debian release because Debian never shipped that version; bullseye had 0.9.8 and bookworm had 1.0.6.)
thoughts on "use less efficient rules" vs "build images with a very specific nft version"?
build images with nft 1.0.1 seems the best option to me , because it seems that nft < 1.0.1 is not prod ready based on the different behaviors we are observing. If this happens between minors in 1.x.y then I think is a different discussion
The question wasn't just about what version we should use in our images, it was how we should deal with the bad interactions with older versions of nftables. It's one thing to say "nftables kube-proxy will not work when using nft < 1.0.1", but it's another thing to say "you are not allowed to run nftables kube-proxy on hosts where anybody is using nft < 1.0.1 (because otherwise kube-proxy will break other users of nft)". There is no reliable way for us to figure out the version of nft on the host from within the kube-proxy container. The best we could do would be to assume that the nft and kernel versions are roughly contemporary, and use the kernel version as a proxy. nft 1.0.1 was contemporary with kernel 5.16, and nft 1.0.0 was contemporary with 5.13. If we require kernel 5.13 / nft 1.0.1, we can support distros newer than mid 2021: RHEL 9 (kernel 5.14 + backports / nft 1.0.4) Debian 12 / bookworm / newstable (kernel 6.1 / nft 1.0.6) Ubuntu 22.04LTS (kernel 5.15 or newer / nft 1.0.2) (Not sure about SLES?) Recent Fedora and other non-LTS distros Whereas our current plan of requiring kernel 5.4 / nft 0.9.2 means "newer than late 2019" and adds: RHEL 8 (kernel 4.18 + many backports / originally nft 0.9.3, later 1.0.4) Debian 11 / bullseye / oldstable (kernel 5.10 / nft 0.9.8) Ubuntu 20.04LTS (kernel 5.4 or newer / nft 0.9.3) (Not sure about SLES?) OK, so first decision: do we feel like we need to support running nftables kube-proxy on RHEL 8 / Debian 11 / Ubuntu 20.04LTS, or are we OK with only supporting the newer distros? For comparison, seems to be leaning toward 4.18/4.19 (from late 2018) as the minimum supported kernel for kubernetes as a whole, so requiring kernel 5.13 for nftables would add several more years before we could really even consider making it the default backend for kube-proxy. (We could also have kube-proxy require the newer versions by default, but have a config option to allow the older versions, and document exactly what can go wrong with that.) If we do want to still support the older distros (either unconditionally or via config option), there are two more questions: we decouple our kernel and nft requirements, and say that kube-proxy requires nft = 1.0.1 regardless of what kernel you're using? (I think "yes"?) we want to do some sort of workaround for the segfault bug (either changing the endpoint rules, or shipping the kube-proxy image with nft == 1.0.2 rather than whatever-version-is-currently-in-debian)? (If we had a config option to allow older kernels, we could make that option also control whether we used vmap-style or iptables-style endpoint rules?)
OK, leaning toward: Docs: We document that nftables kube-proxy requires nftables = 1.0.1, and recommends a kernel = 5.13 We also document that, currently, it can be used on systems with kernels as old as 5.4 if you set the option in the kube-proxy config, but (a) this will likely cause problems with other nftables users on the system (with details), and (b) this option is intended to go away at some point as the LTS distros with older kernels become unsupported. Code: We add to We check the nft client version (via a new knftables API), and if it's < 1.0.1, we error out. We check the kernel version (via ) and if it's < 5.13: if is (the default) we error out if is : if an with a expression passes, we continue, after warning that we might break other users on the system if the fails, we error out because the kernel is too oldWe don't change our rules or change the version of nft in our images, because the problems will only affect users who have explicitly opted in.
oh, and CI apparently has a 5.15 kernel, so we won't need to update the kind job config
I envision we'll hit more of these things in the future, so it will be nice you implement it, in a modular way, kind of the iptables runner that has all these different behaviors hidden from the users
The idea was that for things that affect knftables's own functioning, or its public API, it would check those things itself and hide the details from the caller. But the caller would be expected to use to figure out things like "can I use in a rule", since knftables can't just abstract around that (in general). (Right now the only thing knftables abstracts around is whether table/chain/set/map comments are supported; you're always allowed to specify them in the API, but .) I was originally thinking that the "require nft 1.0.1" restriction would be at the kube-proxy level rather than the knftables level, but I'm thinking now it would make more sense to require that for all knftables users.

Discussion:
"encoding/base32" "fmt" "net" <ins> "os" </ins> "reflect" "strconv" "strings"

discovery "k8s.io/api/discovery/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/version" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/tools/events" "k8s.io/klog/v2"

"k8s.io/kubernetes/pkg/proxy/metrics" proxyutil "k8s.io/kubernetes/pkg/proxy/util" "k8s.io/kubernetes/pkg/util/async" <ins> utilkernel "k8s.io/kubernetes/pkg/util/kernel" </ins> utilexec "k8s.io/utils/exec" netutils "k8s.io/utils/net" "k8s.io/utils/ptr"

) (*Proxier, error) { logger := klog.LoggerWithValues(klog.FromContext(ctx), "ipFamily", ipFamily) <ins> nft, err := getNFTablesInterface(ipFamily) if err != nil { return nil, err } </ins> if initOnly { logger.Info("System initialized and --init-only specified") return nil, nil

serviceHealthServer := healthcheck.NewServiceHealthServer(hostname, recorder, nodePortAddresses, healthzServer) <del> var nftablesFamily knftables.Family if ipFamily == v1.IPv4Protocol { nftablesFamily = knftables.IPv4Family } else { nftablesFamily = knftables.IPv6Family } nft, err := knftables.New(nftablesFamily, kubeProxyTable) if err != nil { return nil, err } </del> proxier := &Proxier{ ipFamily:            ipFamily, svcPortMap:          make(proxy.ServicePortMap),

return proxier, nil } <ins> // Create a knftables.Interface and check if we can use the nftables proxy mode on this host. func getNFTablesInterface(ipFamily v1.IPFamily) (knftables.Interface, error) { var nftablesFamily knftables.Family if ipFamily == v1.IPv4Protocol { nftablesFamily = knftables.IPv4Family } else { nftablesFamily = knftables.IPv6Family } // We require (or rather, knftables.New does) that the nft binary be version 1.0.1 // or later, because versions before that would always attempt to parse the entire // nft ruleset at startup, even if you were only operating on a single table. // That's bad, because in some cases, new versions of nft have added new rule // types in ways that triggered bugs in older versions of nft, causing them to // crash. Thus, if kube-proxy used nft < 1.0.1, it could potentially get locked // out of its rules because of something some other component had done in a // completely different table. nft, err := knftables.New(nftablesFamily, kubeProxyTable) if err != nil { return nil, err } // Likewise, we want to ensure that the host filesystem has nft >= 1.0.1, so that // it's not possible that *our* rules break *the system's* nft. (In particular, we // know that if kube-proxy uses nft >= 1.0.3 and the system has nft <= 0.9.8, that // the system nft will become completely unusable.) Unfortunately, we can't easily // figure out the version of nft installed on the host filesystem, so instead, we // check the kernel version, under the assumption that the distro will have an nft // binary that supports the same features as its kernel does, and so kernel 5.13 // or later implies nft 1.0.1 or later. https://issues.k8s.io/122743 // // However, we allow the user to bypass this check by setting // `KUBE_PROXY_NFTABLES_SKIP_KERNEL_VERSION_CHECK` to anything non-empty. if os.Getenv("KUBE_PROXY_NFTABLES_SKIP_KERNEL_VERSION_CHECK") != "" { kernelVersion, err := utilkernel.GetVersion() if err != nil { return nil, fmt.Errorf("could not check kernel version: %w", err) } if kernelVersion.LessThan(version.MustParseGeneric(utilkernel.NFTablesKubeProxyKernelVersion)) { return nil, fmt.Errorf("kube-proxy in nftables mode requires kernel %s or later", utilkernel.NFTablesKubeProxyKernelVersion) } } return nft, nil } </ins> // internal struct for string service information type servicePortInfo struct { *proxy.BaseServicePortInfo

const UserNamespacesSupportKernelVersion = "6.3" const TmpfsNoswapSupportKernelVersion = "6.4" <ins> // NFTablesKubeProxyKernelVersion is the lowest kernel version kube-proxy supports using // nftables mode with by default. This is not directly related to any specific kernel // commit; see https://issues.k8s.io/122743#issuecomment-1893922424 const NFTablesKubeProxyKernelVersion = "5.13" </ins>