Change:
(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem. This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign

Discussion:
} nodeIP := detectNodeIP(client, hostname, config.BindAddress) <del> protocol := utiliptables.ProtocolIPv4 if utilsnet.IsIPv6(nodeIP) { klog.V(0).Infof("kube-proxy node IP is an IPv6 address (%s), assume IPv6 operation", nodeIP.String()) protocol = utiliptables.ProtocolIPv6 } else { klog.V(0).Infof("kube-proxy node IP is an IPv4 address (%s), assume IPv4 operation", nodeIP.String()) } iptInterface = utiliptables.New(execer, protocol) </del> <ins> klog.Infof("Detected node IP %s", nodeIP.String()) </ins> // Create event recorder eventBroadcaster := record.NewBroadcaster()

klog.V(2).Info("DetectLocalMode: '", string(detectLocalMode), "'") <ins> primaryProtocol := utiliptables.ProtocolIPv4 if utilsnet.IsIPv6(nodeIP) { primaryProtocol = utiliptables.ProtocolIPv6 } iptInterface = utiliptables.New(execer, primaryProtocol) var ipt [2]utiliptables.Interface dualStack := utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) && proxyMode != proxyModeUserspace if dualStack { // Create iptables handlers for both families, one is already created // Always ordered as IPv4, IPv6 if primaryProtocol == utiliptables.ProtocolIPv4 { ipt[0] = iptInterface ipt[1] = utiliptables.New(execer, utiliptables.ProtocolIPv6) // Just because the feature gate is enabled doesn't mean the node // actually supports dual-stack if _, err := ipt[1].ChainExists(utiliptables.TableNAT, utiliptables.ChainPostrouting); err != nil { klog.Warningf("No iptables support for IPv6: %v", err) dualStack = false } } else { ipt[0] = utiliptables.New(execer, utiliptables.ProtocolIPv4) ipt[1] = iptInterface } } if dualStack { klog.V(0).Infof("kube-proxy running in dual-stack mode, %s-primary", iptInterface.Protocol()) } else { klog.V(0).Infof("kube-proxy running in single-stack %s mode", iptInterface.Protocol()) } </ins> if proxyMode == proxyModeIPTables { klog.V(0).Info("Using iptables Proxier.") if config.IPTables.MasqueradeBit == nil {

return nil, fmt.Errorf("unable to read IPTables MasqueradeBit from config") } <del> if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) { </del> <ins> if dualStack { </ins> klog.V(0).Info("creating dualStackProxier for iptables.") <del> // Create iptables handlers for both families, one is already created // Always ordered as IPv4, IPv6 var ipt [2]utiliptables.Interface if iptInterface.IsIPv6() { ipt[1] = iptInterface ipt[0] = utiliptables.New(execer, utiliptables.ProtocolIPv4) } else { ipt[0] = iptInterface ipt[1] = utiliptables.New(execer, utiliptables.ProtocolIPv6) } </del> // Always ordered to match []ipt var localDetectors [2]proxyutiliptables.LocalTrafficDetector localDetectors, err = getDualStackLocalDetectorTuple(detectLocalMode, config, ipt, nodeInfo)

proxymetrics.RegisterMetrics() } else if proxyMode == proxyModeIPVS { klog.V(0).Info("Using ipvs Proxier.") <del> if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) { </del> <ins> if dualStack { </ins> klog.V(0).Info("creating dualStackProxier for ipvs.") <del> // Create iptables handlers for both families, one is already created // Always ordered as IPv4, IPv6 var ipt [2]utiliptables.Interface if iptInterface.IsIPv6() { ipt[1] = iptInterface ipt[0] = utiliptables.New(execer, utiliptables.ProtocolIPv4) } else { ipt[0] = iptInterface ipt[1] = utiliptables.New(execer, utiliptables.ProtocolIPv6) } </del> nodeIPs := nodeIPTuple(config.BindAddress) // Always ordered to match []ipt

return nil } <ins> func (f *fakeIPTables) ChainExists(tableName utiliptables.Table, chainName utiliptables.Chain) (bool, error) { _, _, err := f.getChain(tableName, chainName) if err != nil { return false, err } return true, nil } </ins> // Returns index of rule in array; < 0 if rule is not found func findRule(chain *fakeChain, rule string) int { for i, candidate := range chain.rules {

FlushChain(table Table, chain Chain) error // DeleteChain deletes the specified chain.  If the chain did not exist, return error. DeleteChain(table Table, chain Chain) error <ins> // ChainExists tests whether the specified chain exists, returning an error if it // does not, or if it is unable to check. ChainExists(table Table, chain Chain) (bool, error) </ins> // EnsureRule checks if the specified rule is present and, if not, creates it.  If the rule existed, return true. EnsureRule(position RulePosition, table Table, chain Chain, args ...string) (bool, error) // DeleteRule checks if the specified rule is present and, if so, deletes it.

// Poll until stopCh is closed or iptables is flushed err := utilwait.PollUntil(interval, func() (bool, error) { <del> if exists, err := runner.chainExists(tables[0], canary); exists { </del> <ins> if exists, err := runner.ChainExists(tables[0], canary); exists { </ins> return false, nil } else if isResourceError(err) { klog.Warningf("Could not check for iptables canary %s/%s: %v", string(tables[0]), string(canary), err)

// so we don't start reloading too soon. err := utilwait.PollImmediate(iptablesFlushPollTime, iptablesFlushTimeout, func() (bool, error) { for i := 1; i < len(tables); i++ { <del> if exists, err := runner.chainExists(tables[i], canary); exists || isResourceError(err) { </del> <ins> if exists, err := runner.ChainExists(tables[i], canary); exists || isResourceError(err) { </ins> return false, nil } }

} } <del> // chainExists is used internally by Monitor; none of the public Interface methods can be // used to distinguish "chain exists" from "chain does not exist" with no side effects func (runner *runner) chainExists(table Table, chain Chain) (bool, error) { </del> <ins> // ChainExists is part of Interface func (runner *runner) ChainExists(table Table, chain Chain) (bool, error) { </ins> fullArgs := makeFullArgs(table, chain) runner.mu.Lock() defer runner.mu.Unlock() <del> trace := utiltrace.New("iptables Monitor CANARY check") </del> <ins> trace := utiltrace.New("iptables ChainExists") </ins> defer trace.LogIfLong(2 * time.Second) _, err := runner.run(opListChain, fullArgs)

return nil } <ins> // ChainExists is part of iptables.Interface func (*FakeIPTables) ChainExists(table iptables.Table, chain iptables.Chain) (bool, error) { return true, nil } </ins> // EnsureRule is part of iptables.Interface func (*FakeIPTables) EnsureRule(position iptables.RulePosition, table iptables.Table, chain iptables.Chain, args ...string) (bool, error) { return true, nil