Change:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.
I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.
Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on

Discussion:
// the mark-for-drop chain KubeMarkDropChain utiliptables.Chain = "KUBE-MARK-DROP" <ins> // the kubernetes forward chain kubeForwardChain utiliptables.Chain = "KUBE-FORWARD" </ins> ) // IPTablesVersioner can query the current iptables version.

} } <ins> // Unlink the forwarding chain. args = []string{ "-m", "comment", "--comment", "kubernetes forwarding rules", "-j", string(kubeForwardChain), } if err := ipt.DeleteRule(utiliptables.TableFilter, utiliptables.ChainForward, args...); err != nil { if !utiliptables.IsNotFoundError(err) { glog.Errorf("Error removing pure-iptables proxy rule: %v", err) encounteredError = true } } </ins> // Flush and remove all of our chains. iptablesData := bytes.NewBuffer(nil) if err := ipt.SaveInto(utiliptables.TableNAT, iptablesData); err != nil {

encounteredError = true } } <del> { filterBuf := bytes.NewBuffer(nil) writeLine(filterBuf, "*filter") writeLine(filterBuf, fmt.Sprintf(":%s - [0:0]", kubeServicesChain)) writeLine(filterBuf, fmt.Sprintf("-X %s", kubeServicesChain)) writeLine(filterBuf, "COMMIT") </del> <ins> // Flush and remove all of our chains. iptablesData = bytes.NewBuffer(nil) if err := ipt.SaveInto(utiliptables.TableFilter, iptablesData); err != nil { glog.Errorf("Failed to execute iptables-save for %s: %v", utiliptables.TableFilter, err) encounteredError = true } else { existingFilterChains := utiliptables.GetChainLines(utiliptables.TableFilter, iptablesData.Bytes()) filterChains := bytes.NewBuffer(nil) filterRules := bytes.NewBuffer(nil) writeLine(filterChains, "*filter") for _, chain := range []utiliptables.Chain{kubeServicesChain, kubeForwardChain} { if _, found := existingFilterChains[chain]; found { chainString := string(chain) writeLine(filterChains, existingFilterChains[chain]) writeLine(filterRules, "-X", chainString) } } writeLine(filterRules, "COMMIT") filterLines := append(filterChains.Bytes(), filterRules.Bytes()...) </ins> // Write it. <del> if err := ipt.Restore(utiliptables.TableFilter, filterBuf.Bytes(), utiliptables.NoFlushTables, utiliptables.RestoreCounters); err != nil { </del> <ins> if err := ipt.Restore(utiliptables.TableFilter, filterLines, utiliptables.NoFlushTables, utiliptables.RestoreCounters); err != nil { </ins> glog.Errorf("Failed to execute iptables-restore for %s: %v", utiliptables.TableFilter, err) encounteredError = true }

} } <ins> // Create and link the kube forward chain. { if _, err := proxier.iptables.EnsureChain(utiliptables.TableFilter, kubeForwardChain); err != nil { glog.Errorf("Failed to ensure that %s chain %s exists: %v", utiliptables.TableFilter, kubeForwardChain, err) return } comment := "kubernetes forward rules" args := []string{"-m", "comment", "--comment", comment, "-j", string(kubeForwardChain)} if _, err := proxier.iptables.EnsureRule(utiliptables.Prepend, utiliptables.TableFilter, utiliptables.ChainForward, args...); err != nil { glog.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", utiliptables.TableFilter, utiliptables.ChainForward, kubeForwardChain, err) return } } </ins> // // Below this point we will not return until we try to write the iptables rules. //

} else { writeLine(proxier.filterChains, utiliptables.MakeChainLine(kubeServicesChain)) } <ins> if chain, ok := existingFilterChains[kubeForwardChain]; ok { writeLine(proxier.filterChains, chain) } else { writeLine(proxier.filterChains, utiliptables.MakeChainLine(kubeForwardChain)) } </ins> if chain, ok := existingNATChains[kubeServicesChain]; ok { writeLine(proxier.natChains, chain) } else {

"-m", "addrtype", "--dst-type", "LOCAL", "-j", string(kubeNodePortsChain)) <ins> // If the masqueradeMark has been added then we want to forward that same // traffic, this allows NodePort traffic to be forwarded even if the default // FORWARD policy is not accept. writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-m", "comment", "--comment", `"kubernetes forwarding rules"`, "-m", "mark", "--mark", proxier.masqueradeMark, "-j", "ACCEPT", ) // The following rules can only be set if clusterCIDR has been defined. if len(proxier.clusterCIDR) != 0 { // The following two rules ensure the traffic after the initial packet // accepted by the "kubernetes forwarding rules" rule above will be // accepted, to be as specific as possible the traffic must be sourced // or destined to the clusterCIDR (to/from a pod). writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-s", proxier.clusterCIDR, "-m", "comment", "--comment", `"kubernetes forwarding conntrack pod source rule"`, "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", "-j", "ACCEPT", ) writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-m", "comment", "--comment", `"kubernetes forwarding conntrack pod destination rule"`, "-d", proxier.clusterCIDR, "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", "-j", "ACCEPT", ) } </ins> // Write the end-of-table markers. writeLine(proxier.filterRules, "COMMIT") writeLine(proxier.natRules, "COMMIT")

ChainPrerouting  Chain = "PREROUTING" ChainOutput      Chain = "OUTPUT" ChainInput       Chain = "INPUT" <ins> ChainForward     Chain = "FORWARD" </ins> ) const (