jiebi/CodeConvo huggingface dataset:
Code change:
return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

Related discussion:
text: This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .
Code change:
return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

Related discussion:
text: /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.
Code change:
return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

Related discussion:
text: I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.
Code change:
return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

Related discussion:
text: I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?
Code change:
return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

Related discussion:
text: If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.
Code change:
return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

Related discussion:
text: From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
if e.Decorator != nil { e.Decorator(out) } <ins> if dryrun.IsDryRun(options.DryRun) { if err := dryrun.ResetMetadata(obj, out); err != nil { return nil, err } } </ins> return out, nil }

Related discussion:
text: Running kubectl create with dry-run returns unwanted uid metadata field, as well as a generated name when using generateName. This is inconsistent with the expectations as defined in which states I expected uid, and a generated name to be empty when using the dry-run create option. Expected metadata: Actual metadata: No response <detailsI've tested this from 1.19.0 through 1.23.0 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
reopening, is reverted in to make sure requests with both name+generateName are handled correctly
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .
Code change:
if e.Decorator != nil { e.Decorator(out) } <ins> if dryrun.IsDryRun(options.DryRun) { if err := dryrun.ResetMetadata(obj, out); err != nil { return nil, err } } </ins> return out, nil }

Related discussion:
text: /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
kube := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kube) stdout, err = exec.Command("sudo", "systemctl", "restart", kube).CombinedOutput() <del> if err == nil { return } framework.Failf("Failed to restart kubelet with systemctl: %v, %v", err, stdout) </del> <ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) Eventually(func() ([]string, error) { return filepath.Glob("/var/lib/kubelet/device-plugins/nvidiaGPU*.sock") }, 5*time.Minute, framework.Poll).ShouldNot(ConsistOf(beforeSocks)) </ins> } func getDeviceId(f *framework.Framework, podName string, contName string, restartCount int32) string { // Wait till pod has been restarted at least restartCount times. Eventually(func() bool { p, err := f.PodClient().Get(podName, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> if err != nil || len(p.Status.ContainerStatuses) < 1 { return false } </ins> return p.Status.ContainerStatuses[0].RestartCount >= restartCount <del> }, time.Minute, time.Second).Should(BeTrue()) </del> <ins> }, 5*time.Minute, framework.Poll).Should(BeTrue()) </ins> logs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, podName, contName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", podName, err)

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.8 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): COS Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node /priority important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
FYI, the started to pass now with the merged PR. Will keep watching to make sure it is not flaky.
Code change:
onRelease     = make(chan struct{}) lockObj runtime.Object <ins> gets    int </ins> updates int <ins> wg      sync.WaitGroup </ins> ) <ins> resetVars := func() { onNewLeader = make(chan struct{}) onRenewCalled = make(chan struct{}) onRenewResume = make(chan struct{}) onRelease = make(chan struct{}) </ins> <del> resourceLockConfig := rl.ResourceLockConfig{ Identity:      "baz", EventRecorder: &record.FakeRecorder{}, } c := &fake.Clientset{} c.AddReactor("get", objectType, func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { if lockObj != nil { return true, lockObj, nil } return true, nil, errors.NewNotFound(action.(fakeclient.GetAction).GetResource().GroupResource(), action.(fakeclient.GetAction).GetName()) }) // create lock c.AddReactor("create", objectType, func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { lockObj = action.(fakeclient.CreateAction).GetObject() return true, lockObj, nil }) c.AddReactor("update", objectType, func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { updates++ // Second update (first renew) should return our canceled error // FakeClient doesn't do anything with the context so we're doing this ourselves if updates == 2 { close(onRenewCalled) <-onRenewResume return true, nil, context.Canceled } else if updates == 3 { close(onRelease) } lockObj = action.(fakeclient.UpdateAction).GetObject() return true, lockObj, nil }) c.AddReactor("*", "*", func(action fakeclient.Action) (bool, runtime.Object, error) { t.Errorf("unreachable action. testclient called too many times: %+v", action) return true, nil, fmt.Errorf("unreachable action") }) lock, err := rl.New(objectType, "foo", "bar", c.CoreV1(), c.CoordinationV1(), resourceLockConfig) if err != nil { t.Fatal("resourcelock.New() = ", err) </del> <ins> lockObj = nil gets = 0 updates = 0 </ins> } lec := LeaderElectionConfig{ <del> Lock:          lock, </del> LeaseDuration: 15 * time.Second, RenewDeadline: 2 * time.Second, RetryPeriod:   1 * time.Second,

Related discussion:
text: The lock acquired by tryAcquireOrRenew is released when the leader ends leadership. However, due to the cancellation of the context, the lock may be set as an empty lock, so the Update cannot be run normally, resulting in a failure to release the lock. The lock can be released every time the leader ends. run this: and then you can got a log like these: and you when you watch the leases at the same time: you can see that was failed to release because of No response <details</details<details</details<details</details<details</details<details</detailsrelated packages: v1.3.0 v0.25.0 v0.25.0 v2.70.1
/sig api-machinery
/triage accepted thanks!
/unassign
/assign
/unassign
/unassign me