Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In Ubuntu 18.04 DNS queries are being served via daemon which is listening on and serves queries only from . The only line in is , and this file is managed by systemd. Once any pod with notation appears on the host, fires the iptables rule in chain of table: -A POSTROUTING -s 127.0.0.0/8 -o lo -m comment --comment "SNAT for localhost access to hostports" -j MASQUERADE See commit and lines for details. Therefore all queries to goes not from , but from interface with default route due to masquerading, and rejects all of these requests with systemd-resolved[]: Got packet on unexpected IP range, refusing. How to reproduce it (as minimally and precisely as possible): Just install Kubernetes with CNI on Ubuntu-18.04 (with kubeadm, kubespray, chef-kubernetes, whatever) and run any pod with . Voila! You just lost DNS resolving. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Digital Ocean (but reproduceable on any configuration) OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: (but reproduceable with any install tool) Others: CNI: 0.6.0 CNI plugins: 0.7.1 CNI daemon: weave /sig network
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hi, is anyone managed to figure a workaround for this?
Don't use Ubuntu 18.04?
Please if you have nothing useful to add, just move along. The workaround that I have settled on, is to run the following command on your Ubuntu 18.04 system during your bootstrapping: This ^^ solves the issue for me.
Can you explain what that means? Thank you!
Adding an SNAT rule above the rule generated by kubelet fixes it (replace the with whatever the slot is above the offending rule for you): I'm going to look at upstreaming this fix.
Thanks for your solution! Is it possible to fix on systemd side?
reopen if needed
but it is still broken, why did you close issue?
I have the same issue on Kubernetes version 1.14.2 on Ubuntu 18.04
Fixed in
Bug still present in kubernetes 1.16.0
I don't know if people still encounter this issue. For kops users, the solution I've found is to add this to cluster spec: This will actually tell kubelets to use the systemd insted of the default one in . For the users not using kops, you can just set in the kublet options.
I'm still having it on Ubuntu 19.10 running latest 1.6.2 minikube (aka k8s v1.17.0) but with "cri-o" as container engine which enables CNI by default. Since cri-o is very slow with releasing .deb packages - it is 1.15 which may be still impacted by this issue. Adding to solves issue - but it does not solve issue described in here. Temporarily I solve it by running: after first DaemonSet starts first Pod.
Update: After updating cri-o from 1.15 to 1.16 the problem vanished.

Discussion:
return fmt.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", tc.table, tc.chain, kubeHostportsChain, err) } } <del> // Need to SNAT traffic from localhost args = []string{"-m", "comment", "--comment", "SNAT for localhost access to hostports", "-o", natInterfaceName, "-s", "127.0.0.0/8", "-j", "MASQUERADE"} if _, err := iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting, args...); err != nil { return fmt.Errorf("Failed to ensure that %s chain %s jumps to MASQUERADE: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, err) </del> <ins> if natInterfaceName != "" && natInterfaceName != "lo" { // Need to SNAT traffic from localhost args = []string{"-m", "comment", "--comment", "SNAT for localhost access to hostports", "-o", natInterfaceName, "-s", "127.0.0.0/8", "-j", "MASQUERADE"} if _, err := iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting, args...); err != nil { return fmt.Errorf("Failed to ensure that %s chain %s jumps to MASQUERADE: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, err) } </ins> } return nil }

// Add implements port mappings. // id should be a unique identifier for a pod, e.g. podSandboxID. // podPortMapping is the associated port mapping information for the pod. <del> // natInterfaceName is the interface that localhost used to talk to the given pod. </del> <ins> // natInterfaceName is the interface that localhost uses to talk to the given pod, if known. </ins> Add(id string, podPortMapping *PodPortMapping, natInterfaceName string) error // Remove cleans up matching port mappings // Remove must be able to clean up port mappings without pod IP