Change:
I am running kubernetes 0.5.2 + and I've noticed that the number of entries in iptables increases every few seconds. I started with a clean install, no pods or replicationControllers, there is now two services by default (kubernetes-ro and kubernetes). I ran and got . Next, I turned up 7 services and 3 replicationControllers which resulted in 3 pods also being turned up. I started a watch... I saw "38", "43, "50, "64", "66", "68", "70", ..., "115" and on until I stopped . It appears the proxy continues to add REDIRECT's for the services, for example the lines that keep being look like: A similar line for each service gets every second or so, it appears to all be in the table. I'm thinking the newly function might be the culprit, since calls and in the log I see every few seconds (looks like does a sync every 5 seconds).
I noticed that the 0.5.3 and 0.5.4 releases don't include the , though it's in master. Perhaps the issue I've filed is already known and I missed it when searching the issues.
I have not heard this issue. Can you paste a listing of iptables-save -t nat when you are seeing this problem? Alternatively, I can cut you a branch with some extra debug instrumentation if you're willing to test it out today...
I'm not able to test it out today. As far as the iptables output, I'd have to type it in (can't copy/paste -- different physical network). However, the output has lines, like the one given in the issue, which are every 5 seconds to the "nat" table. It seems like the check to see if the rule already exists is perhaps not functioning properly. I am working on a fix for (because it uses the iptables -C flag which isn't supported in older iptables), in that fix I created a function to check if an existing rule already exists: Example call: Using may not be ideal, perhaps breaking up the output of the iptables command into a slice and checking each line would be better(?) sure yet.
Fix is pending - would be great if someone could verify it against an actually old iptables (I just faked the version string).
should be able to try it out friday, maybe monday
Feel free to rip off our checks with out echeck function :) On Nov 26, 2014 7:36 PM, "Craig Wickesser" wrote:
that it actually works. On Nov 26, 2014 7:40 PM, "Tim Hockin" wrote:
I fired up kuberenetes 0.5.2 (plus + your changes) and things looked good initially (when starting up kubernetes, no pods/services, etc). The number of entries iptables-save remained constant. I then turned up some services and watched the length of iptables-save increase. I turned down the services and the number of entries in iptables-save continues to increase. At the moment, every 5 seconds it's back to adding two entries (one for the kubernetes service and one for the kubernetes-ro service). With log level set to 0, I see "Opened iptables portal for service.." messages in the proxy log every 5 seconds ().
Are you sure you have the right proxy version running? I could repro the initial problem, but not this - it works just fine. To be clear when you say it increases after creating a service - do you mean it increases by 1 rule or that it increases continually? , Craig Wickesser wrote:
On Monday/Tuesday I'll try the latest from master, perhaps I messed something up. To clarify, the increase was continuous. When I initially setup kubernetes (with a fresh/clean etcd), things looked normal with regards to iptables. I then deployed some pods and services and kept watching iptables-save. The number of entries increased every 5 seconds. I deleted pods, replication controllers and services and the number of entries continued to increase, but only by two (for the kubernetes and kubernetes-ro services) every 5 seconds.
Did you paste the output of iptables-save? Its possible that it is slightly different from my version. This is why parsing text is no fun :( On Nov 28, 2014 3:42 PM, "Craig Wickesser" wrote:
I did not include the output of iptables-save. I'll see if I can get additional output besides the line included in the initial issue.
Just a couple lines so I can see the format. On Nov 28, 2014 7:32 PM, "Craig Wickesser" wrote:
here's a few lines from : All of the lines follow the exact same format. Looks like the proxier (to fit with the output I provided) would need to look like this:
FWIW - I'm using CentOS 6.5 with iptables version 1.4.7.
I recompiled with changes to proxier as mentioned earlier (changed the order of args) and did not resolve the issue. I continue seeing (in the proxy log):
I've some additional logging to log the and the . I am logging them using these two lines: When I reduce the output between the and the (when it should return "true", that fields is a superset), the result looks like: Notice has quotes around it in the set, and the does not? I think perhaps that's the issue and that's why the check is false.
I just tried changing to: and that's working. Basically, strip out quotes from the line. There might be a better alternative.
Uggh, seriously? Your iptables add quotes around the comment, mine omits the quotes for single tokens. I guess I can force it to not be a single word, something like "service: " service name. What a hack. Any better ideas? , Craig Wickesser wrote:
Hmm, it seems worse than that. If I feed in a comment with quotes, I get ""foo"", so I can't do that to be compatible with your implementation. Can you try something? iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment "foo1" iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment '"foo2"' iptables-save | grep foo -A PREROUTING -d 1.2.3.4/32 -m comment --comment ""foo2"" -A PREROUTING -d 1.2.3.4/32 -m comment --comment foo1 , Tim Hockin wrote:
i'll try that in about 8 hours :)
here's what I got:
ugh, that is SO BROKEN. I'm going to have to do something really unpleasant. , Craig Wickesser wrote:
I have a potential fix, but airport wifi prevents me from pushing git <shrug. will publish it later.
sounds good, I appreciate the effort.
PR posted - please test before we commit.
I've tried the PR and a comment to it.
got the same problem here using Kubernetes v0.6.0-12-gd927031aecdeb6 on redhat 6.5
It looks like 0.6 was cut before - - can we patch that in, or should we wait of 0.7? , polariss0i wrote:
got the same problem on kubernetes v1.0.3 and iptables-v1.4.7. Should I file a new issue?

Discussion:
return false, fmt.Errorf("error checking rule: %v", err) } <del> // Sadly, iptables has inconsistent quoting rules for comments. // Just unquote any arg that is wrapped in quotes. argsCopy := make([]string, len(args)) copy(argsCopy, args) for i := range argsCopy { unquote(&argsCopy[i]) </del> <ins> // Sadly, iptables has inconsistent quoting rules for comments. Just remove all quotes. // Also, quoted multi-word comments (which are counted as a single arg) // will be unpacked into multiple args, // in order to compare against iptables-save output (which will be split at whitespace boundary) // e.g. a single arg('"this must be before the NodePort rules"') will be unquoted and unpacked into 7 args. var argsCopy []string for i := range args { tmpField := strings.Trim(args[i], """) argsCopy = append(argsCopy, strings.Fields(tmpField)...) </ins> } argset := util.NewStringSet(argsCopy...)

// Check that this is a rule for the correct chain, and that it has // the correct number of argument (+2 for "-A <chain name>") <del> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(args)+2 { </del> <ins> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(argsCopy)+2 { </ins> continue } // Sadly, iptables has inconsistent quoting rules for comments. <del> // Just unquote any arg that is wrapped in quotes. </del> <ins> // Just remove all quotes. </ins> for i := range fields { <del> unquote(&fields[i]) </del> <ins> fields[i] = strings.Trim(fields[i], """) </ins> } // TODO: This misses reorderings e.g. "-x foo ! -y bar" will match "! -x foo -y bar"

return false, nil } <del> func unquote(strp *string) { if len(*strp) >= 2 && (*strp)[0] == '"' && (*strp)[len(*strp)-1] == '"' { *strp = strings.TrimPrefix(strings.TrimSuffix(*strp, `"`), `"`) } } </del> // Executes the rule check using the "-C" flag func (runner *runner) checkRuleUsingCheck(args []string) (bool, error) { out, err := runner.run(opCheckRule, args)