jiebi/CodeConvo huggingface dataset:
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.
Code change:
} } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {

Related discussion:
text: From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
fi if ! _out="$(diff -Naupr -x "BUILD" -x "OWNERS" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor ${_kubetmp}/vendor)"; then <del> echo "Your vendored results are different:" echo "${_out}" echo "Godeps Verify failed." </del> <ins> echo "Your vendored results are different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2 </ins> echo "${_out}" > vendordiff.patch <del> echo "If you're seeing this locally, run the below command to fix your directories:" echo "patch -p0 < vendordiff.patch" echo "(The above output can be saved as godepdiff.patch if you're not running this locally)" echo "(The patch file should also be exported as a build artifact if run through CI)" </del> <ins> echo "If you're seeing this locally, run the below command to fix your directories:" >&2 echo "patch -p0 < vendordiff.patch" >&2 echo "(The above output can be saved as godepdiff.patch if you're not running this locally)" >&2 echo "(The patch file should also be exported as a build artifact if run through CI)" >&2 </ins> KEEP_TMP=true if [[ -f vendordiff.patch && -d "${ARTIFACTS_DIR:-}" ]]; then echo "Copying patch to artifacts.."

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output
Code change:
// TODO: move it to k8s.io/utils/net, this is the same as current AddIPOffset() // but using netip.Addr instead of net.IP func addOffsetAddress(address netip.Addr, offset uint64) (netip.Addr, error) { <del> addressBig := big.NewInt(0).SetBytes(address.AsSlice()) r := big.NewInt(0).Add(addressBig, big.NewInt(int64(offset))) addr, ok := netip.AddrFromSlice(r.Bytes()) </del> <ins> addressBytes := address.AsSlice() addressBig := big.NewInt(0).SetBytes(addressBytes) r := big.NewInt(0).Add(addressBig, big.NewInt(int64(offset))).Bytes() // r must be 4 or 16 bytes depending of the ip family // bigInt conversion to bytes will not take this into consideration // and drop the leading zeros, so we have to take this into account. lenDiff := len(addressBytes) - len(r) if lenDiff > 0 { r = append(make([]byte, lenDiff), r...) } else if lenDiff < 0 { return netip.Addr{}, fmt.Errorf("invalid address %v", r) } addr, ok := netip.AddrFromSlice(r) </ins> if !ok { <del> return netip.Addr{}, fmt.Errorf("invalid address %v", r.Bytes()) </del> <ins> return netip.Addr{}, fmt.Errorf("invalid address %v", r) </ins> } return addr, nil }

Related discussion:
text: We ran into an issue with an old version of k8s 1.18 that we were running where subnet for cluster IP is given as . The control plane comes up just fine but then when you try to create a k8s service, it fails due to a broken behavior with the IP Allocator Since this is not really a supported version of the k8s, I went around digging up a bit more to see what is the regular behavior and found this to be very inconsistent across a few release branches. bash go test -timeout 30m -run ^TestAllocateNextFC$ -v === RUN TestAllocateNextFC ? --- PASS: TestAllocateNextFC (0.00s) PASS ok 0.566s go func TestAllocateNextFC(t *testing.T) { , cidr, err := netutils.ParseCIDRSloppy("fc::/112") if err != nil { t.Fatal(err) } // Need to be replaced with the suitable handler for that version of the branch r, err := newTestAllocator(cidr) if err != nil { t.Fatal(err) } defer r.Destroy() ip, err := r.AllocateNext() if err != nil { t.Fatalf("%s", err.Error()) } fmt.Println(ip.String()) } Add the above Test case under the IP allocator test case No response_ <detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details
/sig network /assign /kind bug
Thanks for helping me with this issue.
/triage accepted /assign
Code change:
"${DOCKER[@]}" rm -f -v "$1" >/dev/null 2>&1 || true } <ins> # Validate a release version # # Globals: #   None # Arguments: #   version # Returns: #   If version is a valid release version # Sets: #  BASH_REMATCH, so you can do something like: #    local -r version_major="${BASH_REMATCH[1]}" #    local -r version_minor="${BASH_REMATCH[2]}" #    local -r version_patch="${BASH_REMATCH[3]}" function kube::release::parse_and_validate_release_version() { local -r version_regex="^v(0|[1-9][0-9]*).(0|[1-9][0-9]*).(0|[1-9][0-9]*)$" local -r version="${1-}" [[ "${version}" =~ ${version_regex} ]] || { kube::log::error "Invalid release version: '${version}'" return 1 } } </ins> # --------------------------------------------------------------------------- # Building

Related discussion:
text: (cc Despite doing very, very similar things, doesn't rely on . It should, or the logic should otherwise be consolidated. I'm going to maybe work on this to make easier.