jiebi/CodeConvo huggingface dataset:
Code change:
// in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

Related discussion:
text: I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.
Code change:
// in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

Related discussion:
text: I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?
Code change:
// in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

Related discussion:
text: If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.
Code change:
// in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

Related discussion:
text: From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
Stop() // Returns a chan which will receive all the events. If an error occurs <del> // or Stop() is called, this channel will be closed, in which case the // watch should be completely cleaned up. </del> <ins> // or Stop() is called, the implementation will close this channel and // release any resources used by the watch. </ins> ResultChan() <-chan Event }

Related discussion:
text: The docs for the Watch interface are a little ambiguous: "If an error occurs or Stop() is called, this channel will be closed, in which case the watch should be completely cleaned up." That can be read two ways: When the channel is closed, the watch is already completely cleaned up When the channel is closed, the caller must call Stop() to clean up resources. If it's the first, can I suggest we rephrase to "If an error occurs or Stop() is called, this channel will be closed. It is not required to call Stop() to free resources when the channel has been closed." If it's the second, can I suggest we rephrase to "If an error occurs or Stop() is called, this channel will be closed. On observing the closed channel, the caller must call Stop() to free resources." /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : The docs for the Watch interface are a little ambiguous: </details
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : The docs for the Watch interface are a little ambiguous: </details
/assign Mike would you be interested in taking a look?
OK, I will take a look.
Code change:
// AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":                   "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions":         "conditions contains information about the current status of the endpoint.", "hostname":           "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":          "targetRef is a reference to a Kubernetes object that represents this endpoint.",

Related discussion:
text: Taking only into consideration the first endpoint which is in the same zone of the node or has no zone dependency and is ready for serving. & Would it make sense to reopen this issue?
I don't follow entirely what you are suggesting, can you elaborate?
kube-proxy should understand which zone a node is in and prefer endpoints in the same zone.
but that is unrelated to this issue, topology is considered See KEPs related, topology-aware-routing was replaced by topology-aware-hints
thank you for clarifying. It seems ambiguous to me to have a zone attribute and a hint attribute. I fail to implement the hinting on a service in v1.23.8, but this is urelated to this issue.
heh, don't worry, this topic is complex and we went through different solutions, you have more historical context on the KEPs I linked above and this thread in the mailing list
Code change:
} // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { </ins> return "" }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Code change:
command := exec.Command("umount", target) output, err := command.CombinedOutput() if err != nil { <del> if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } if mounter.withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", target) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) </del> <ins> return checkUmountError(target, command, output, err, mounter.withSafeNotMountedBehavior) </ins> } return nil }

Related discussion:
text: A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.
Code change:
} } } <ins> func TestFairEvaluationForNodes(t *testing.T) { defer algorithmpredicates.SetPredicatesOrderingDuringTest(order)() predicates := map[string]algorithmpredicates.FitPredicate{"true": truePredicate} numAllNodes := 500 nodeNames := make([]string, 0, numAllNodes) for i := 0; i < numAllNodes; i++ { nodeNames = append(nodeNames, strconv.Itoa(i)) } nodes := makeNodeList(nodeNames) g := makeScheduler(predicates, nodes) // To make numAllNodes % nodesToFind != 0 g.percentageOfNodesToScore = 30 nodesToFind := int(g.numFeasibleNodesToFind(int32(numAllNodes))) // Iterating over all nodes more than twice for i := 0; i < 2*(numAllNodes/nodesToFind+1); i++ { nodesThatFit, _, _, err := g.findNodesThatFit(context.Background(), framework.NewCycleState(), &v1.Pod{}) if err != nil { t.Errorf("unexpected error: %v", err) } if len(nodesThatFit) != nodesToFind { t.Errorf("got %d nodes filtered, want %d", len(nodesThatFit), nodesToFind) } if g.nextStartNodeIndex != (i+1)*nodesToFind%numAllNodes { t.Errorf("got %d lastProcessedNodeIndex, want %d", g.nextStartNodeIndex, (i+1)*nodesToFind%numAllNodes) } } } </ins>

Related discussion:
text: What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help
Code change:
} } <ins> // in older kernels (< 3.18), the destination address family attribute doesn't exist so we must // assume it based on the destination address provided. if d.AddressFamily == 0 { // we can't check the address family using net stdlib because netlink returns // IPv4 addresses as the first 4 bytes in a []byte of length 16 where as // stdlib expects it as the last 4 bytes. addressFamily, err := getIPFamily(addressBytes) if err != nil { return nil, err } d.AddressFamily = addressFamily } </ins> // parse Address after parse AddressFamily incase of parseIP error if addressBytes != nil { ip, err := parseIP(addressBytes, d.AddressFamily)

Related discussion:
text: Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.