Change:
What happened: When creating nodes on machines with kube-proxy is unable initialize and route service traffic. The following is logged: This is compat issue in which I believe is called directly from kube-proxy. This is likely due to module reorganization with iptables move to nftables: iptables 1.8 is backwards compatible with iptables 1.6 modules: However kube-proxy is based off of debian:stretch which iptables-1.8 may only make it to as part of stretch-backports How to reproduce it (as minimally and precisely as possible): Install a node onto a host with iptables-1.8 installed (ex: Debian Testing/Buster) Anything else we need to know?: I can keep these nodes in this config for a while, feel free to ask for any helpful output. Environment: Kubernetes version (use ): PRETTYNAME="Debian GNU/Linux buster/sid" NAME="Debian GNU/Linux" ID=debian HOMEURL="" SUPPORTURL="" BUGREPORTURL="" Linux vm28 4.16.0-1-amd64 SMP Debian 4.16.5-1 (2018-04-29) x86_64 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For the record, this probably breaks at least Calico and Weave as well, based on my abject failures to get pod<pod networking to function on Debian Buster (which has upgraded to iptables 1.8). I'm filing bugs for that now, but this breaking change to iptables may be worth a wider broadcast to the k8s community.
itself seems compatible with iptables =1.8 so the slogan in this issue is somewhat misleading. I have made basic tests and see no problems when using the correct version of the user-space (and ipv6 with ip6tables) and the supporting libs. I don't think this problem can be fixed by altering some code in kube-proxy. Tested versions; iptables v1.8.2, linux 4.19.3 The problem seems to be that that user-space program (and libs) is (and has always been) dependent on the kernel version on the host. When the user-space program is in a container with a old version this problem is bound to happen sooner or later, and it will happen again. The kernel/user-space dependency is one of the problem that is supposed to fix. A long-term solution may be to replace with ntf or bpf.
Iptables v1.8.2 have 2 modes (depending on soft-links); and; seem to work fine with both. BTW; I have not tested any network policies, that is not of course but is is iptables.
While the title is somewhat murky the fact is that is distributed using images based on debian-stretch and pulls in the userspace from that distribution. When those images are run on hosts with a newer iptables this fails. To be clear: this isn't a defect in the code, it's a defect in packaging/release.
Do you mean it breaks on a newer kernel? The iptables binary is part of kube-proxy so what would the on-host iptables have to do with anything? I don't understand.
There are 2 sets of modules for packet filtering in the kernel: iptables, and nftables. Until recently, you controlled the iptables ruleset with the family of tools, and nftables with the tools. In iptables 1.8, the maintainers have "deprecated" the classic iptables: the tool now does userspace translation from the legacy UI/UX, and uses nftables under the hood. So, the commands look and feel the same, but they're now programming a different kernel subsystem. The problem arises when you mix and match invocations of iptables 1.6 (the previous stable) and 1.8 on the same machine, because although they look identical, they're programming different kernel subsystems. The problem is that at least Docker does some stuff with on the host (uncontained), and so you end up with some rules in nftables and some rules (including those programmed by kube-proxy and most CNI addons) in legacy iptables. Empirically, this causes weird and wonderful things to happen - things like if you trace a packet coming from a pod, you see it flowing through both iptables and nftables, but even if both accept the packet, it then vanishes entirely and never gets forwarded (this is the failure mode I reported to Calico and Weave - bug links upthread - after trying to run k8s on debian testing, which now has iptables 1.8 on the host). Bottom line, the networking containers on a machine have to be using the same minor version of the binary as exists on the host.
Do you think it would be sufficient to enforce (if possible) the host version of iptables to "legacy"; and keep the =1.8 version? I build and install myself and the "mode" is determined by a soft-link; I assume the same applies for "Debian Testing/Buster" and others, but I don't knoe for sure.
thanks. That was very succinct. What a crappy situation. How are we to know what is on the host? Can we include BOTH binaries in our images and probe the machine to see if either has been used previously (e.g. lsmod or something in /sys) ?
As a preface, one thing to note: iptables 1.8 ships two binaries, and . The latter always programs iptables. So, there's fortunately no need to bundle two versions of iptables into a container, you can bundle just iptables 1.8 and be judicious about which binary you invoke... At least until the -legacy binary gets deleted, presumably in a future release. Here's some requirements I think an ideal solution would have: k8s networking must continue to function, obviously. should be robust to the host iptables getting upgraded while the system is running (e.g. in the background). should be robust to other k8s pods (e.g. CNI addons) using the "wrong" version of iptables. should be invisible to cluster operators - k8s should just keep working throughout. should not require a "flag day" on which everything must cut over simultaneously. There's too many things in k8s that touch iptables (docker, kube-proxy, CNI addons) to enforce that sanely, and k8s's eventual consistency model doesn't make a hard cutover without downtime possible anyway. at the very least, the problem should be detected and surfaced as a fatal node misconfiguration, so that any automatic cluster healing can attempt to help. So far I've only thought up crappy options for dealing with this. I'll throw them out in the hopes that it leads to better ideas. Mount chunks of the host filesystem (/usr/sbin, /lib, ...) into kube-proxy's VFS, and make it chroot() to that quasi-host-fs when executing iptables commands. That way it's always using exactly the binary present on the host. Introduces obvious complexity, as well as a bunch of security risks if an attacker gets code execution in the kube-proxy container. Using iptables 1.8 in the container, probe both and for the presence of rules installed by the host. Hopefully, there will be rules in only one of the two, and that can tell kube-proxy which one to use. This is subject to race conditions, and is fragile to host mutations that happen after kube-proxy startup (e.g. that upgrades iptables and restarts the docker daemon, shifting its rules over to nftables). Can solve it with periodic reconciling (i.e. "oops, host seems to have switched to nftables, wipe all iptables rules and reinstall them in nftables!") Punt the problem up to kubeadm and an entry in the cluster object. IOW, just document that "it's your responsibility to correctly tell kube-proxy which version of iptables you're using, or things will break." Relies on humans to get things right, which I predict will cause a rash of broken clusters. If we do this, we should absolutely also wire something into node-problem-detector that fires when both iptables and nftables have rules programmed. Have a cutover release in which kube-proxy starts using nftables exclusively, through the tools, and mandate that host OSes for k8s must do everything in nftables, no iptables allowed. Likely intractable given the variety of addons and non-k8s software that does stuff to the firewall (same reason has endured all these years even though nftables is measurably better in every way). Find some kernel hackers and ask them if there's any way to make iptables and nftables play nicer together, so that userspace can just continue tolerating mismatches indefinitely. I'm assuming this is ~impossible, otherwise they'd have done it already to facilitate the transition to nftables. Create a new DaemonSet whose sole purpose is to be an RPC-to-iptables translator, and get all iptables-using pods in k8s to use it instead of talking direct to the kernel. Clunky, expensive, and doesn't solve the problem of host software touching stuff. Just document (via a Sonobuoy conformance test) that this is a big bag of knives, and kick the can over to cluster operators to figure out how to safely upgrade k8s in place given these constraints. I can at least speak on behalf of GKE and say that I sure hope it doesn't come to that, because all our options are strictly worse. I can also speak as the author of MetalLB and say that the support load from people with broken on-prem installs will be completely unsustainable for me :) Of all of these, I think "probe with both binaries and try to conform to whatever is already there" is the most tractable if kube-proxy were the only problem pod... But given the ecosystem of CNI addons and other third-party things, I foresee never ending duels of controllers flapping between iptables and nf_tables endlessly, all trying to vaguely converge on a single stack, but never succeeding.
When using iptables 1.8.2 in nftables mode (my version; v6.38) is still used by kube-proxy. But in ipset is "". It seems to work anyway but I can't understand how, or maybe my testing is insufficient. I will try to test better and make sure the ipset's are used so they are not just defined and not used and my tests just happens to work. But is any one can explain the relation between iptables in mode and please give a reference to some doc.
Ipset is only used in proxy-mode=ipvs. I get hits on ipset rules so they work in some way;
When using nf_tables mode rules are indefinitely to the chain; in both proxy-mode ipvs and iptables.
I experienced the same issue in . As a workaround I found in the oracle docs, which made the pods be able to communicate with each other as well as with the outside world again.
I discussed iptables/nft incompatibility in too, although that was before the iptables binary got rewritten... It seems like for right now, the backward-compatible answer is "you have to make sure the host is using iptables in legacy mode".
FWIW, I hit this issue as well when deploying Kubernetes on Debian Buster. I've included some logging in .
this works for me
You said you have some iptables-experts. Can we get advice on a canonical way to probe the system for the preferred binary to run? If we have that, we can code kube-proxy and kubelet to be smarter
simple answer is to run the host's iptables binaries. The decision whether to use iptables-legacy or iptables-nft depends solely on the binaries you are running. Everything we are talking about is running in the host's network namespace and already has to cooperate with other things running in the host's network namespace, especially in areas touching the firewall.
I read a bit more of the bug here. Is the core problem that the IPVS proxy specifically is using ipset, which has some specific integration with iptables, but when the host has iptables-nft as the iptables binaries that causes problems? My iptables/nftables people say that ipset + iptables-nft should work correctly... What iptables-nft/iptables-legacy versions are people using here that have the problem?
this issue has way too many things in it. The first comment's bug (can't initialize iptables table `nat': Table does not exist (do you need to insmod?)) I believe was fixed some months ago already in iptables upstream.
The bug is "with iptables = 1.8, the host and all containers have to use the same iptables mode or things won't work". Other comments may or may not be relevant to that. Yeah, but how simple is that? It's not just a matter of mounting /usr/sbin/iptables into your container, because that's a symlink to something else. Is just mounting in all of /usr/sbin enough, or do you need to mount in /lib64 (or whatever) as well to guarantee you'll get the right shared libraries? Does it ever look in /etc/sysconfig? Etc. What exactly are we doing in OCP?
the symlink itself will actually tell you what the binary is. Following the 'iptables' symlink will either: 1) be a direct symlink to iptables-legacy or iptables-nft 2) on systems that use 'alternatives' (because hey Linux is all about CHOICE! right???) we can call alternatives to tell us (but I'm not sure exactly what alternatives does underneath): Even that's a bit more complicated. Possibly the best choice here is to simply accept that if a container wants to modify the host OS then it may need to run tools provided by the host OS and not blindly assume that stuff it ships internally can always be used. Mount the host bin/lib/etc into /host and then have your internal /usr/sbin/iptables be a chroot wrapper into those dirs so that when kube-proxy calls iptables it actually runs the wrapper and does the right thing. I remain unconvinced that containers that wish to modify the host OS can just blindly go about whatever they want to do and assume the host OS doesn't matter.
Another option I'm checking is whether the container could call 'nft list <something' to see if iptables-nft had already run before. Not sure yet, but might be accurate enough to let tools the container bundles decide.
I wasn't trying to argue for that. I was just saying that "use the host iptables binary" is a very underspecified answer, and we need to provide more detail than that, or we're going to keep fielding duplicates of this bug forever. (I guess in the general case you have no idea what the underlying distro is so you really can't assume anything beyond " itself is in either or ". So probably the answer is "Mount the node's onto in the container and then into that to run "?)
Yeah, basically. Which is what we're doing in OpenShift. Probably need to mount /lib too because the iptables binaries link to libxtables and libip[4|6]tc and libpcap.
So it was suggested by iptables developers that 'nft list ruleset' could be run by the thing wishing to know whether iptables-legacy or iptables-nft was being used. If that returns anything, it indicates that nftables has been initialized and is being used and thus iptables-nft should be used. If not, that indicates that either (a) iptables-legacy is in-use or (b) nothing has created any netfilter tables yet so the system is in uninitialized state. (b) is pretty unlikely as usually during system bootup things add firewall rules before any containers are run.
Just wanted to post another confirmation here that when installing / running a K8s cluster on Raspberry Pis with Raspbian 10 / Buster, I had to run: Otherwise I was getting lots of errors with networking from various non-k8s-core pods (e.g. coredns, ingress, kube-proxy, kube-apiserver were seemingly fine, but flannel, metrics-server, nfs-client-provisioner were crashlooping). I ran the above command on each node and rebooted all nodes, and everything quickly switched to status.
I can confirm that updating the host to use iptables-legacy works on Raspbian 10 (arm) and Debian 10 (amd64) to resolve the iptables mismatch issue. For completeness it may be beneficial to update all of the network tools to use the legacy versions to avoid issues. These commands may or may not be pertinent depending upon specific host configuration but will avoid mixing legacy and nft modes if invoked from outside docker/kubernetes.
Have we decided on an approach here? While pinning iptables to program iptables is good, it sounds like if we don't win the race to update alternatives, and something else programs an nftable rule, we end up with both iptables and nftables and that packets disappear (per ). But on the other hand, if we can ensure that we don't mix iptables and nftables, do we now have to run all tests with both nftables and iptables as the backing implementation? Is nftables something we feel confident behaves identically? We could do the same thing we did with swap: refuse to start if we detect any nftables usage, because while it's possible to use correctly, all signs point to no. Should we then expedite a move to something like eBPF that gets us above the fray of the "table wars"?
Our approach in OpenShift is to have the relevant pods , and in the we install in /usr/sbin that chroot to the host filesystem and exec the copy of iptables there. These images then work on any system regardless of whether it has old or new iptables, and in the latter case, whether that iptables is configured to use "legacy" or "nft" mode. (In particular, these images work on both RHEL 7, using legacy iptables, and RHEL 8, using new iptables in nft mode.) But I'm thinking we should get iptables upstream to add a new "figure it out" mode to the client binaries, which would internally do something along the lines of what suggested to figure out if the system iptables was using nft mode or legacy mode, and then it would just use the same mode. Then we just tell everyone "make sure your containers are using the package from iptables version 1.8.whatever or later" and they don't have to worry beyond that.
my concern is the validation though - should we now run every e2e once with nftables and once with iptables? (Ignoring more efficient testing strategies) I like your suggestion of having iptables do something more sensible, but even if we could guarantee that we only use nf_tables, I don't believe that's what we've built & tested for.
The two modes are supposed to be equivalent in terms of behavior. (The advantage of using iptables in nft mode is that it lets other parts of the system use nft directly, and get nft's advantages, and their rules will interoperate correctly with the iptables-nft rules. Whereas if you use iptables-legacy, the iptables rules and nft rules would conflict with each other in complicated ways.) So anyway, the two modes are supposed to be equivalent, so we shouldn't have to test against both modes, and if we did, and something in kubernetes didn't work right in one mode, that would indicate an iptables or kernel bug, not a kubernetes bug. It's possible we might end up wanting to add workarounds to kubernetes for a bug in one or the other mode at some point, if someone discovers such a bug, but I don't think we need to be testing against both modes continuously. (And in practice, OCP on RHEL 8 using nft mode works just fine, other than possibly one problem with a mysteriously not actually rejecting and behaving like it was .)
Thanks - good to know it does seem to work on RHEL8! My view is that we should actively choose ebpf vs nft vs ipvs vs ipt, and if someone wants to change that they have to chop the wood and carry the water - i.e. if the iptables maintainers do the work to replace iptables in our containers, and are generally available to fix breakages, then we should consider adopting nft. In the absence of that, my view is that we should make a deliberate decision. If that means we're not yet ready to support nft, we could simply refuse to run when the OS configuration is incompatible, like we did with swap.
That's the approach we've taken with Calico. We found an iptables-nft bug and worked around. The iptables developers were responsive when we reported the bug and, as it happens, they'd already committed a fix for the issue we hit and that fix was released into debian within a few weeks. We haven't decided what we're doing wrt to detecting nft mode. So far, we just require the user to select it with a flag. Some issues with mounting in the host filesystem: people will get annoyed (we've gotten flak for mounting in the kernel modules directory so we can load iptables extensions, for example) the host may not have iptables or it may be installed without optional packages (debian split iptables and iptables v6 IIRC and they probably put all the kernel modules in individual packages because they like to be fine-grained) Telling people to switch to legacy mode won't work for long. Soon, nftables itself will be in active use and legacy mode won't be an option for a lot of folks.
A built-in "figure it out" mode seems right. This is frankly ridiculous. This is what APIs are for, and like it or not is an API. Forcing all parties to coordinate and use the same binaries is ridonculous and clearly not workable. It sounds like openshift has implemented a "figure it out" mode on its own. But I know many customers who are not going to be happy hostPath mounting / into kube-proxy. Do we REALLY need the host's binaries or can we install iptables 1.8+ and call our own which does the same detection? What distros are known to have 1.8 available so I can do some playing?
Debian Buster is the main one (as Debian is used as the default distro by many of the k8s components), Ubuntu 19.04, RHEL 8 (and the upcoming Centos 8 by extension), Alpine 3.10, Fedora = 29.
Ah, I too stumbled on Debian 10 (or buster) for a couple of hours thinking I have messed my networking. The above command did not help in my case; installation through . Everything started working after installing Debian 9.9.0 :)
I think perhaps the core oversight from iptables folks was not thinking that people would want to alter the host netns's firewall both from the host system and simultaneously from within a chroot whose software does not match the host. Outside the container/k8s ecosystem, that would seem like a daft thing to do, so it doesn't surprise me that we're in this mess now :( FWIW, empirically I'm seeing a significant uptick in support requests for MetalLB that end up being caused by this mismatch, so if nothing else, it would be worth calling out as a gotcha in the k8s installation documentation (along with the workaround for Debian-based systems). to your question about the hostPath mounting: unless further changes happen upstream, it'd be fine to install iptables 1.8+ in the container and judiciously invoke the correct variant, rather than mount the host's copy. The only remaining risk there is if bugfixes are only applied to one of the two, and a bugfix involves changing how the firewall is programmed at the netlink layer. Then you could end up with the two iptables fighting each other.
AIUI, there's no obvious way to tell from the kernel which mode is in use on a freshly-booted system. Both subsystems (ipt and nft) have existed for years now, so that's not sufficient. The "signal" is the destination of the symlinks (yes, symlinks - I shudder at the thought of them diverging). We could write a wrapper that uses whichever subsystem already has rules present, but that is terrifyingly racy and definitely violates the least-surprise principle. Then, if there are no rules present, pick one. Coming off of the discussion about SCTP, I wonder if the right way for distros to "signal" which subsystem they wish to use is by module blacklisting. Separately, we've been able to be very lazy around multiple iptables versions, and that party is certainly over. I'm unaware if there are any guarantees between different versions of the nft-ipt-compat logic. Can older versions correctly work with the ruleset generated by newer ones? What happens when kube-proxy uses the host's iptables-nft binary, and an Istio container uses an older (debian) one? What if (ugh) kubelet provided a mini-firewalld?
That is correct, but fortunately we don't have to handle "freshly-booted" systems, we only have to handle systems where kubelet is already running. Kubelet will create iptables rules of its own with the system iptables binaries (we don't support systemd-containerized kubelet any more) before any containers are running, so containers can just say "if there are any rules in nft then use iptables-nft, else use iptables-legacy". That's not a terrible idea but it still requires having both sets of binaries in every container and having a wrapper to figure out which one to use... so it's not that much better than the "see which subsystem has rules" idea.
Your main point stands, but the Debian (and Ubuntu, and ...) packages aren’t fine-grained: contains all the tools, and the kernel package contains all the modules. There are package splits in the source, but they split out library packages, not tool packages, and the package depends on them all anyway.
Looks like it would be good to document any workarounds for 1.16 release notes since folks are hitting this already? (see for example)
Perhaps Kubernetes could bind mount a static iptables "proxy" binary into containers (at all popular locations) that proxies to the host's iptables. Update: To clarify, per OOB request, I'm not proposing bind mounting a directory or even any of the host's files/binaries. I'm proposing we write a new static binary (let's call it ) in, say, Go, and then whenever we start a container we bind mount at all possible paths that and friends are commonly seen at in various distros/containers. When invoked, forwards its arguments and environment (or necessary subset) off to a bind-mounted Unix socket or link local service running on the host, communicating with a new host-side daemon/service that executes the iptables/etc command on the host (using the host's preferred kernel mechanism) and replies with the stdout/stderr/exit code back to the client. Because it'd be a static binary, we don't need to deal with spraying potentially-conflicting shared libraries from the host all over the containers. We'd shadow just the iptables binaries, for all containers that have one of those binaries at one of those paths.
This sounds very feasible. I started digging a bit deeper. First bump: our image builds on which builds on , which is stretch. We could a) convert everything to buster as debian-base v2.0.0; b) start publishing debian version-named base images; c) one-off this one. thoughts? It's unclear the scope of such a change. We'd want something in-path replacing ; ; . We'd want logic akin to: Does that seem right? I tested iptables-nft and observed that it creates nftables in family .
for release viz.
also cc
Trying to do a naive debian-base update is somewhat fraught. Step 7/11 : RUN echo "Yes, do as I say!" | apt-get purge bash e2fsprogs libcap2-bin libmount1 libsmartcols1 libudev1 libblkid1 libss2 libsystemd0 ncurses-base ncurses-bin tzdata ---Running in Reading package lists... Building dependency tree... Reading state information... Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: apt : Depends: libapt-pkg5.0 (= 1.7.0~alpha3~) but it is not going to be installed Recommends: ca-certificates but it is not going to be installed E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages. The command '/bin/sh -c echo "Yes, do as I say!" | apt-get purge bash e2fsprogs libcap2-bin libmount1 libsmartcols1 libudev1 libblkid1 libss2 libsystemd0 ncurses-base ncurses-bin tzdata' returned a non-zero code: 100 make[1]: [Makefile:78: build] Error 100 make[1]: Leaving directory '/usr/local/google/home/thockin/src/k/kubernetes/build/debian-base' make: [Makefile:56: sub-build-amd64] Error 2
Removing and make that go away. claims to not exist. A bunch of others claim to not be installed. Streamlining that list gets me a build of 52.4MB there's apt magic happening that I don't follow, and your fingerprints are on it :)
Now an error installing nftables: dpkg (subprocess): unable to execute new nftables package pre-installation script (): No such file or directory dpkg: error processing archive /tmp/apt-dpkg-install-4N16Ls/5-nftables0.9.0- (--unpack): new nftables package pre-installation script subprocess returned error exit status 2 dpkg (subprocess): unable to execute new nftables package post-removal script (): No such file or directory dpkg: error while cleaning up: new nftables package post-removal script subprocess returned error exit status 2 Errors were encountered while processing: /tmp/apt-dpkg-install-4N16Ls/5-nftables0.9.0- E: Sub-process /usr/bin/dpkg returned an error code (1)
it looks like the nftables pkg wants bash
It's not entirely reliable. The default tables will be created any time you run iptables-nft, so if the user is on an iptables-legacy system but runs a stray pod that erroneously invokes iptables-nft, then that would create those tables and then cause your wrapper to do the wrong thing until reboot. We eventually decided that you can't really do any better than: legacylines=$(iptables-legacy-save 2/dev/null echo 0) nftlines=$(iptables-nft-save 2/dev/null echo 0) if [ "{nftlines}" ]; then mode=nft else mode=legacy fi which is terrible, but it's unlikely to be fooled. You'd definitely want to only do this once and then cache the answer. I had started working on this at but the scripts and README there are all totally inconsistent with each other because I kept changing my mind about what the best way to make this work was...
If I push a branch, can you start integrating with that, maybe PR you changes into my branch? has another idea he will expand on in a but, but I want to see how bad this rabbithole gets. NB: We'd need every single container that uses iptables to participate in this... We'd also need some EOL plan - when can we stop doing this?
Updated my comment above:
is a WIP on building base images. We'd need to hook into the Dockerfile to take over the bare iptables commands
We discussed a bit offline, but for the record, some thoughts: We would have to do this on every pod (also maybe consider net capabilities). That doesn't seem SO bad - it's a small set. Do we have to handle RuntimeClass that isn't cgroups-based? Probably not going to intersect with . I don't think we need to handle non-privileged pods (or without the caps we care about) - they should not be able to use iptables anyway, but we should triple check that. What about things that run iptables inside their own netns (e.g. istio's capture)? do they need to use the same backend, or can it be mixed mode at that scope? FYI We would have to run this new daemonset on every node. Need telemetry, provisioning, etc. That should be a wash wrt actual memory consumption, but only if we can reclaim some from kube-proxy and/or calico and/or ... This plays very badly with PodSecurityPolicy unless we trap in WAY below it (at CRI) which is a much harder fix. Doing it as admission is more transparent, but possibly subject to ordering bugs.
Friendly ping from 1.16 release lead. I wanted to let you know that we are planning to cut 1.16.0-rc.1 tomorrow and go into code-thaw. Please let me know if this fix needs to be considered as 1.16 release blocking.
Every single container that uses iptables in the root network namespace. It's fine for, eg, istio, to use whatever iptables mode it wants in the pod namespace. (Though if you have multiple sidecars in a pod they all need to use the same mode...) Probably as long as we care about people running Kubernetes on RHEL/CentOS 7. (People will probably be running RHEL 7 longer than people are running CentOS 7, but we might care about those users less. Either way, by the time we stop caring about that, everyone else should be using nft mode.) That is correct. Pods need to be hostNetwork and either privileged or for them to matter.
The only thing we can do in the near term is tell people to use legacy mode. Even 1.8.2 (as present in debian-buster) is broken. How best to document this?
Do we do anything even remotely similar to this currently? (The "overwriting binaries in other people's containers without telling them" part, not the proxying part.) It's nice in that it solves the problem for everyone all at once but... not nice in every other way :slightlysmilingface: Random idea halfway between the two current approaches: add a new volume type "" or "" or something, and if you mount a volume of that type into your pod, you'll find that it contains iptables binaries that do the right thing via unspecified means. (And in the future, maybe also contains other binaries to solve similar host/pod interaction issues? Kind of a more powerful downward API sort of thing.)
I agree it's kind of awful, but so is the problem... We'd STILL need to run that daemonset, which is a significant change.
We should be able to give containers a working set of iptables-legacy or iptables-nft binaries directly rather than needing a proxy. Just give them an entire chroot rather than just the binaries. (ie, build a Debian container image containing only the package and the packages it depends on (eg, glibc), and then mount that somewhere in the pod). Then instead of overwriting their with a proxy binary, you overwrite it with a shell script that does , etc. Or that works with the volume idea too; the volume would just contain the chroot within it in addition to the wrapper scripts.
Agreed. I proposed it because it'd fix everything all at once, rather than waiting for all network add-ons to update to a work either the old way (just exec iptables) or some new way (find some new downward API directory to chroot into). How long would that take? Put it in the kubelet? Too gross? It's already in the business of calling iptables, no?
Reminder (as this discussion shakes out): if you're calling , there is a nonzero chance you will need to load a module. So, it has become part of the lore that best-practice -callers already always need to bind-mount from the host. If we're going down the route of half-magic bind-mounts, then I think I'd rather see the kubelet assemble it out of bind-mounts from the host, rather than needing a specific container.
No, there isn't. Kubelet will always have created iptables rules before starting any pods. I thought about that, but the reason we didn't try to do that before is that there are no safe assumptions you can make about what the distro-installed version of iptables does and doesn't need from the host filesystem. (eg, there's no a priori reason to think that would be needed, since the iptables source code itself does not refer to any such thing.) If you want to use the system iptables you have to mount the entire filesystem.
if you're calling iptables, there is a nonzero chance you will need to load a module. Each match and action type has its own module, loaded on demand. So if you do then you'll trigger loading of the module. If you do , then you'll load and so on.
There are two different kind of kernel module loading here: the kind that Casey was worrying about is that if you run and it finds that the (or ) module is not loaded, then it will explicitly invoke to load it, and that only works if and are available. For the thing you're talking about, the iptables binary isn't what loads the module. When you send a rule to the kernel using , the kernel netfilter code will decide that it needs to have the module, and so it will send a request to userspace, and that request gets received and handled by in the root net/pid/mount/etc namespace, regardless of where the original iptables call came from. So the container doesn't need access to the modules in that case.
I've created to add documentation to the page. There is on whether we should add this information in other places too. Adding to release notes is already on the agenda:
For future readers not able to make kube-proxy work for some reasons, you might want to look for a replacement: (it does the job plus some other stuff, please take a look at the documentation before doing anything)
Kind of a creepy idea, but you could use nsenter to run the iptables command on the host in the hosts environment.
I've started to play with CentOS 8, it comes with but without Haven't found how OpenShift 4 works yet
The iptables user space tools are provided in a container
The iptables 1.8.2 in RHEL/CentOS 8 has the necessary bugfixes from 1.8.3 backported
The status of this issue is that it was resolved by , if I read correctly, and therefore Kubernetes 1.17 (kube-proxy in 1.17) should work without having to switch the nodes to iptables-legacy?
The official kubernetes packages, and in particular kubeadm-based installs, are fixed as of 1.17. Other distributions of kubernetes may have been fixed earlier or might not be fixed yet.
did we wind up backporting this at all?
no... we'd have to backport the whole rebasing-the-images-to-debian-buster thing which seems like a big change for a point release
We'd really only need to backport using a new image for kube-proxy and maybe hyperkube. On Mon, Mar 16, 2020, 19:06 Dan Winship <:
Updated link:
To get rid of that libvirt error, my permanent workaround in Debian 11 (as a host) with libvirtd daemon is to block the loading of iptables-related modules: Create a file in : daemon now starts without any error. Post-analysis: Apparently, I had module loaded alongside with many nft-related modules; once was gone, the pesky error message went away.

Discussion:
# $1 - server architecture kube::build::get_docker_wrapped_binaries() { local arch=$1 <del> local debian_base_version=v1.0.0 local debian_iptables_version=v11.0.2 </del> <ins> local debian_base_version=v2.0.0 local debian_iptables_version=v12.0.1 </ins> ### If you change any of these lists, please also update DOCKERIZED_BINARIES ### in build/BUILD. And kube::golang::server_image_targets local targets=(

# Several utilities (e.g. ping) were kept for usefulness, but may be removed in later versions. RUN echo "Yes, do as I say!" | apt-get purge  bash  <del> debconf-i18n  e2fslibs  </del> e2fsprogs  <del> init  initscripts  </del> libcap2-bin  <del> libkmod2  </del> libmount1  libsmartcols1  <del> libudev1  </del> libblkid1  <del> libncursesw5  libprocps6  libslang2  </del> libss2  <del> libsystemd0  libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl  </del> ncurses-base  ncurses-bin  <del> systemd  systemd-sysv  sysv-rc  </del> tzdata # No-op stubs replace some unnecessary binaries that may be depended on in the install process (in

IMAGE ?= $(REGISTRY)/debian-base BUILD_IMAGE ?= debian-build <del> TAG ?= v1.0.0 </del> <ins> TAG ?= v2.0.0 </ins> TAR_FILE ?= rootfs.tar ARCH?=amd64

export DOCKER_CLI_EXPERIMENTAL := enabled ifeq ($(ARCH),amd64) <del> BASEIMAGE?=debian:stretch </del> <ins> BASEIMAGE?=debian:buster-slim </ins> endif ifeq ($(ARCH),arm) <del> BASEIMAGE?=arm32v7/debian:stretch </del> <ins> BASEIMAGE?=arm32v7/debian:buster-slim </ins> QEMUARCH=arm endif ifeq ($(ARCH),arm64) <del> BASEIMAGE?=arm64v8/debian:stretch </del> <ins> BASEIMAGE?=arm64v8/debian:buster-slim </ins> QEMUARCH=aarch64 endif ifeq ($(ARCH),ppc64le) <del> BASEIMAGE?=ppc64le/debian:stretch </del> <ins> BASEIMAGE?=ppc64le/debian:buster-slim </ins> QEMUARCH=ppc64le endif ifeq ($(ARCH),s390x) <del> BASEIMAGE?=s390x/debian:stretch </del> <ins> BASEIMAGE?=s390x/debian:buster-slim </ins> QEMUARCH=s390x endif

FROM BASEIMAGE <ins> # Install latest iptables package from buster-backports RUN echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list;  apt-get update;  apt-get -t buster-backports -y --no-install-recommends install iptables # Install other dependencies and then clean up apt caches </ins> RUN clean-install  conntrack  ebtables  ipset  <del> iptables  </del> kmod  netbase <ins> # Install iptables wrapper scripts to detect the correct iptables mode # the first time any of them is run COPY iptables-wrapper /usr/sbin/iptables-wrapper RUN update-alternatives  --install /usr/sbin/iptables iptables /usr/sbin/iptables-wrapper 100  --slave /usr/sbin/iptables-restore iptables-restore /usr/sbin/iptables-wrapper  --slave /usr/sbin/iptables-save iptables-save /usr/sbin/iptables-wrapper RUN update-alternatives  --install /usr/sbin/ip6tables ip6tables /usr/sbin/iptables-wrapper 100  --slave /usr/sbin/ip6tables-restore ip6tables-restore /usr/sbin/iptables-wrapper  --slave /usr/sbin/ip6tables-save ip6tables-save /usr/sbin/iptables-wrapper </ins>

REGISTRY?="staging-k8s.gcr.io" IMAGE=$(REGISTRY)/debian-iptables <del> TAG?=v11.0.2 </del> <ins> TAG?=v12.0.1 </ins> ARCH?=amd64 ALL_ARCH = amd64 arm arm64 ppc64le s390x TEMP_DIR:=$(shell mktemp -d) <del> BASEIMAGE?=k8s.gcr.io/debian-base-$(ARCH):v1.0.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/debian-base-$(ARCH):v2.0.0 </ins> # This option is for running docker manifest command export DOCKER_CLI_EXPERIMENTAL := enabled

<ins> #!/bin/sh # Copyright 2019 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -e # Detect whether the base system is using iptables-legacy or # iptables-nft. This assumes that some non-containerized process (eg # kubelet) has already created some iptables rules. # Bugs in iptables-nft 1.8.3 may cause it to get stuck in a loop in # some circumstances, so we have to run the nft check in a timeout. To # avoid hitting that timeout, we only bother to even check nft if # legacy iptables was empty / mostly empty. num_legacy_lines=$( (iptables-legacy-save || true; ip6tables-legacy-save || true) 2>/dev/null | grep '^-' | wc -l) if [ "${num_legacy_lines}" -ge 10 ]; then mode=legacy else num_nft_lines=$( (timeout 5 sh -c "iptables-nft-save; ip6tables-nft-save" || true) 2>/dev/null | grep '^-' | wc -l) if [ "${num_legacy_lines}" -ge "${num_nft_lines}" ]; then mode=legacy else mode=nft fi fi update-alternatives --set iptables "/usr/sbin/iptables-${mode}" > /dev/null update-alternatives --set ip6tables "/usr/sbin/ip6tables-${mode}" > /dev/null # Now re-exec the original command with the newly-selected alternative exec "$0" "$@" </ins>

# Use go get -u github.com/estesp/manifest-tool to find these values _DEBIAN_BASE_DIGEST = { <del> "manifest": "sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde", "amd64": "sha256:8ccb65cd2dd7e0c24193d0742a20e4a673dbd11af5a33f16fcd471a31486866c", "arm": "sha256:3432b41de3f6dfffdc1386fce961cfd1f9f8e208b3a35070e10ef3e2a733cb17", "arm64": "sha256:9189251e1d1eb4126d6e6add2e272338f9c8a6a3db38863044625bca4b667f31", "ppc64le": "sha256:50aa659e1e75e4231ee8293c3b4115e5755bb0517142b9b4bddbc134bf4354db", "s390x": "sha256:bbb8ee3a2aaca738c00809f450233d98029fea4e319d8faaa30aa94c8b17a806", </del> <ins> "manifest": "sha256:ebda8587ec0f49eb88ee3a608ef018484908cbc5aa32556a0d78356088c185d4", "amd64": "sha256:d7be39e143d4e6677a28c81c0a84868b40800fc979dea1848bb19d526668a00c", "arm": "sha256:fc731da13b0bc9013b85a86b583fc92e50869b5bc8e7aa6ca730ec0240954c7d", "arm64": "sha256:12502c3eed050fa9b6d5fe353a44bfc5f437dc325c8912b1a48dcc180df36f1e", "ppc64le": "sha256:4277aa59b63c5a1369e6d84a295ecc4ffa08985dcf114de9f7b6de1af4fcbc86", "s390x": "sha256:78ef2a6b017539379c1654b4e52ba8519bfec821c62d0b3a1dbd15104b711e21", </ins> } _DEBIAN_IPTABLES_DIGEST = { <del> "manifest": "sha256:b522b0035dba3ac2d5c0dbaaf8217bd66248e790332ccfdf653e0f943a280dcf", "amd64": "sha256:adc40e9ec817c15d35b26d1d6aa4d0f8096fba4c99e26a026159bb0bc98c6a89", "arm": "sha256:58e8a1d3b187eed2d8d3664cd1c9723e5029698714a24dfca4b6ef42ea27a9d4", "arm64": "sha256:1a63fdd216fe7b84561d40ab1ebaa0daae1fc73e4232a6caffbd8353d9a14cea", "ppc64le": "sha256:9f90adbc7513cc96d92fcec7633c4b29e766dd31cf876af03c0b54374e22fa9c", "s390x": "sha256:4f147708deff2a0163ee49b6980cc95423514bec5f4091612d65773b898fbdae", </del> <ins> "manifest": "sha256:d1cd487e89fb4cba853cd3a948a6e9016faf66f2a7bb53cb1ac6b6c9cb58f5ed", "amd64": "sha256:852d3c569932059bcab3a52cb6105c432d85b4b7bbd5fc93153b78010e34a783", "arm": "sha256:c10f01b414a7cd4b2f3e26e152c90c64a1e781d99f83a6809764cf74ecbc46c3", "arm64": "sha256:5725e6fde13a6405cf800e22846ebd2bde24b0860f1dc3f6f5f256f03cfa85bd", "ppc64le": "sha256:b6d6e56a0c34c0393dcba0d5faaa531b92e5876114c5ab5a90e82e4889724c5a", "s390x": "sha256:39e67e9bf25d67fe35bd9dcb25367277e5967368e02f2741e0efd4ce8874db14", </ins> } _DEBIAN_HYPERKUBE_BASE_DIGEST = {

digest = _digest(_DEBIAN_BASE_DIGEST, arch), registry = "k8s.gcr.io", repository = "debian-base", <del> tag = "0.4.1",  # ignored, but kept here for documentation </del> <ins> tag = "v2.0.0",  # ignored, but kept here for documentation </ins> ) container_pull(

digest = _digest(_DEBIAN_IPTABLES_DIGEST, arch), registry = "k8s.gcr.io", repository = "debian-iptables", <del> tag = "v11.0.2",  # ignored, but kept here for documentation </del> <ins> tag = "v12.0.1",  # ignored, but kept here for documentation </ins> ) container_pull(