Code change:
If another process is altering iptables rules at the time k8s attempts to add iptables rules it will fail with a locking error. k8s should use -w and retry after a delay. Below, an external process (openshift-sdn-node) modifies iptables at roughly the same time and k8s fails without retrying. Related to
Sorry, I misunderstood the -w flag, with it retry logic wouldn't be necessary as it'd wait. Perhaps simply retrying is a better solution as -w flag may not exist everywhere we'd like.
Looks like -w was to iptables 1.4.20(August 2013).
Yeah, we have to be pretty back-compatible with iptables. We always retry after a few seconds - is there really a bug here? , Scott Dodson wrote:
It definitely seems like there is some sort of a problem. We see this randomly on system startup. In our environment OpenShift is launching kubernetes around the same time other systemd units are modifying the firewall config. If one of these others units has the lock kubernetes will report the error Scott mentions and then will never create the required nat'ing for services. Here's how to reproduce this: At this point if you run you will see the exact same error kubernetes reports. It's probably not obvious why you have to set the breakpoint at 257 but if you set the breakpoint at 260 and then step through the code it appears that the compiler has optimized lines 263 and 257 to be the same. In any case running will verify if you have the lock or not. Once you've verified you have the lock just restart OpenShift (running kubernetes v0.14.1-582-gb12d75d): At this point you can press in gdb to continue the iptables invocation and release the lock. Afterwards I will see output from cadvisor every once in a while, however the will show the firewall is still blank: I initially restarted Kubernetes with a blank firewall at 11:21 and yet at 11:38 I still have nothing:
I probably should have made it more clear, I removed the lock by pressing 'c' in gdb to continue. That's when I started waiting. I'll update my comment to make that more clear.
So something for a holding the lock? I am out of your office for the rest of this week, but maybe you can figure out - is there a stale process holding a lock? What kind of lock is it? Flock or semaphore or ... ? On Apr 29, 2015 8:44 AM, "Brenton Leanhardt" wrote:
What we're saying is that on startup it's fairly common for multiple systemd units to make firewall additions. If one of them grabs the lock (rarely for more than a few seconds) it will cause kubernetes to fail to initialize iptables and it will never fix itself. The lock only need be grabbed the microsecond kubernetes tries to initialize iptables for this bug to present itself. We know exactly which program in our environment was concurrently making iptables updates. The problem is that kubernetes doesn't appear to be retrying if it fails to initialize iptables.
On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the flag to as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform.
It does retry all up tables ops. Unless it somehow gets pathologically phase locked with your agent, or there is some deeper bug. We have e2e tests that prove you can delete tables and it will recreate them. Are you saying that it never retries (did you set -v=4?) or that every retry fails? On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the -w flag to iptables as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform. — Reply to this email directly or view it on GitHub .
I can grab the lock for just the second that kubernetes needs it and then let it go. The kube-proxy nat'ing will never be rebuilt without intervention (restarting kubernetes). That seems like a bug to me and is what my original comment reproduces. In addition, once the environment is in this state it won't even pick up new service nat rules. Other nodes in my environment will pick them up but this one is dead. I highly suspect it has to do with the fact that iptables initialization is failing. We test all the time that once a system is up and running you can clean iptables and it will come back. This is a slightly different case.
This seems like a v1 blocker - we see this in every realistic environment we put the Kube-proxy in. Disagreement?
I still don't understand this issue. It is demonstrably true that the kube-proxy retries all of the iptables logic periodically. It's possible (given the mention of "won't even pick up new service nat rules") that this locking tickles a bug in kube-proxy, but I am having trouble reproducing it. Can you run your kube-proxy with -v=4, trigger the bug, and then take a look at the logs? If this is real, yeah it's a 1.0 blocker.
The problem is in the initialization process. If you hold the lock while the kube-proxy tries to initialize itself then it will be wedged forever. What you're showing is the kube-proxy can repair the iptables rules after the initialization code has run. The logs I pasted earlier show that after the initialization fails at it will never try again. I tend to run at loglevel 4 by default but I'll try to find time today to run through this again.
I have a fix coming - can you test the PR before commit?
I ran the exact same test as mentioned in my original comment. While the debug lines have changed the end result is the same: grab the lock (at this point even will fail) start the kube-proxy release the lock (at this point works) wait.. try creating another service wait.. The rules never come back. Here are the logs:
Sure, I'll test the PR. I'm betting it will apply fairly cleanly on to my OpenShift codebase. If not I'll launch Kubernetes directly.
Many of those log lines are not part of upstream. In fact, what upstream has is a glog.Fatalf() which will kill kube-proxy, and when it restarts it will try again. Did you guys break it by not actually exiting?
Looks like we did. We need to put it into a restart loop. ----- Original Message -----
is teh fix, though I assert we should not need it, and if you need it this fix may not help you anyway. I am still OK to commit this.
Although NewProxier sucks because it doesn't return a usable error. I'll take the todo to clean up that part. ----- Original Message -----
hey, it logs its error - nothing wrong with that :) , Clayton Coleman wrote:
:)
Opened and will move the rest back to openshift.
I restarted docker and it worked then

Related discussion:
text: }}, }}) <del> p := CreateProxier(lb, net.ParseIP("0.0.0.0"), &fakeIptables{}, net.ParseIP("127.0.0.1")) </del> <ins> p := createProxier(lb, net.ParseIP("0.0.0.0"), &fakeIptables{}, net.ParseIP("127.0.0.1")) </ins> waitForNumProxyLoops(t, p, 0) svcInfoP, err := p.addServiceOnPort(serviceP, "TCP", 0, time.Second)